Web Scraping with Pandas and Beautifulsoup

APIs are not always available. Sometimes you have to scrape data from a webpage yourself. Luckily the modules Pandas and Beautifulsoup can help!

Scripting is where Python really pays off. The Python Scripting Essentials Bundle has the kind of small automation projects you would build for work.

When a site has no API but the data sits in a table, you can scrape the table and put it straight into a pandas DataFrame. From there you can sort, filter and plot it like any other dataset.

Web scraping

Pandas has a neat concept known as a DataFrame. A DataFrame can hold data and be easily manipulated. We can combine Pandas with Beautifulsoup to quickly get data from a webpage.

If you find a table on the web like this:

We can convert it to JSON with:

 
import pandas as pd
import requests
from bs4 import BeautifulSoup

res = requests.get("http://www.nationmaster.com/country-info/stats/Media/Internet-users")
soup = BeautifulSoup(res.content,'lxml')
table = soup.find_all('table')[0] 
df = pd.read_html(str(table))
print(df[0].to_json(orient='records'))
 

And in a browser get the beautiful json output:

Converting to lists

Rows can be converted to Python lists.
We can convert it to a dataframe using just a few lines:

 
import pandas as pd
import requests
from bs4 import BeautifulSoup

res = requests.get("http://www.nationmaster.com/country-info/stats/Media/Internet-users")
soup = BeautifulSoup(res.content,'lxml')
table = soup.find_all('table')[0] 
df = pd.read_html(str(table))[0]
countries = df["COUNTRY"].tolist()
users = df["AMOUNT"].tolist()
 

Pretty print pandas dataframe

You can convert it to an ascii table with the module tabulate.
This code will instantly convert the table on the web to an ascii table:

 
import pandas as pd
import requests
from bs4 import BeautifulSoup
from tabulate import tabulate

res = requests.get("http://www.nationmaster.com/country-info/stats/Media/Internet-users")
soup = BeautifulSoup(res.content,'lxml')
table = soup.find_all('table')[0] 
df = pd.read_html(str(table))
print( tabulate(df[0], headers='keys', tablefmt='psql') )
 

Download web scraping examples

The steps

• Download the page with urllib.request.urlopen().
• Find the table with soup.find('table').
• Pull the rows with table.findAll('tr').
• Pull the cells with row.findAll(['td', 'th']).
• Build a list of lists and hand it to pd.DataFrame().

pd.read_html(url) does the same in one line and works when the page is a plain table. It is worth trying first.

Add a User-Agent header, check the status code and put a small delay between requests. Scraping too fast is the easiest way to get blocked.

Want more practice on this? Practise this on PyChallenge, there are short browser exercises you can run right after reading.