Beautiful soup python web scraping how to#

You need a JavaScript engine in order to do that. Unfortunately, the only way to get the page after JavaScript has ran is, well, running the JavaScript. It's a very frequent problem in my courses to see this happen. Therefore, you would only see the initial page. When we make a request for page content using requests, the JavaScript does not run. After JavaScript runs, it has the data we want.

Here's an example page right before JavaScript kicks in: Example page () right before JavaScript runs.Īnd here it is a few milliseconds after, once JavaScript has kicked in: Example page () right after JavaScript has finished running.Īs you can see, the initial page (before JavaScript runs), has no data of interest to us. Sometimes you will see the initial page content (before the JavaScript runs) for a few moments, and then the JavaScript kicks in. and voilà! We have the information we need and we can feed it to our programs.Ī key difference between loading the page using your browser and getting the page contents using requests is that your browser executes any JavaScript code that the page comes with. So we load the HTML using the requests mode, and parse it using BeautifulSoup. CSS is used to perform styling, and our scraping programs don't care what the page looks like. That's because the HTML usually contains all the information in the page. When we are doing web scraping, all we're interested in is the HTML.

That's usually some HTML code, some CSS, and some JavaScript. When you load up a website you want to scrape using your browser, the browser will make a request to the page's server to retrieve the page content. However, sometimes the scraping doesn't quite work.

Beautiful soup python web scraping how to#

In my courses I teach you how to build web scraping systems very quickly and easily by using two popular Python libraries: requests and beautifulsoup4. Or maybe, when you make your requests, the page content is very different from what you see in your browser? Do you have some scraping code that works for some sites but not others, even though you're sure your method to get data out of the site is working well?