We can download data using the urllib2 module.. These examples work with both http, https and for any type of files including text and image.

SQL is the language you use to ask a database questions. If you want to get better at it, SQLCrab runs real queries in the browser, nothing to install.

Downloading a file in Python is three steps: open the url, read the response and write the bytes to disk. The urllib.request module in the standard library does all of it, so you do not need requests for this.

Data will be saved in the same directory as your program resides. This means if you run your program from C:\apps\, images will be saved there too unless you explicitly specify another directory.

Download data to variable
To download data to a variable, you can use this code:

  
import urllib2
response = urllib2.urlopen('https://wordpress.org/plugins/about/readme.txt')
data = response.read()
print(data)

The first line, ‘import urllib2’, loads the module. The second line opens the connection to the url. The method response.read() downloads data from the url and stores it into the variable data.

Download text file
To download a file you can use this code:

  
import urllib2
response = urllib2.urlopen('https://wordpress.org/plugins/about/readme.txt')
data = response.read()
filename = "readme.txt"
file_ = open(filename, 'w')
file_.write(data)
file_.close()

Download image file
Downloading an image from the web works in the same way:

  
import urllib2
response = urllib2.urlopen('https://upload.wikimedia.org/wikipedia/commons/thumb/f/f8/Python_logo_and_wordmark.svg/260px-Python_logo_and_wordmark.svg.png')
data = response.read()
filename = "image.png"
file_ = open(filename, 'w')
file_.write(data)
file_.close()

Download network examples

The code

  • urllib.request.urlopen(url) opens the connection and gives you a response object.
  • response.read() gives you the content as bytes.
  • Write it with open(filename, 'wb') and f.write(content). Use 'wb' for binary, 'w' is for text.
  • Build the filename from the url, os.path.basename(urlparse(url).path) gives you the last part.

Add a timeout so a slow server cannot freeze your program: urlopen(url, timeout=10). A timeout raises an error you can catch.

Check response.status before you write, and a 404 will save you a file full of html error text.

If you want to try this yourself, the exercises on PyChallenge turn the same idea into a few short problems with instant feedback.