Download file

We can download data using the urllib2 module.. These examples work with both http, https and for any type of files including text and image.

Data will be saved in the same directory as your program resides. This means if you run your program from C:\apps\, images will be saved there too unless you explicitly specify another directory.

Download data to variable
To download data to a variable, you can use this code:

import urllib2
response = urllib2.urlopen('https://wordpress.org/plugins/about/readme.txt')
data = response.read()
print(data)

The first line, ‘import urllib2’, loads the module. The second line opens the connection to the url. The method response.read() downloads data from the url and stores it into the variable data.

Download text file
To download a file you can use this code:

import urllib2
response = urllib2.urlopen('https://wordpress.org/plugins/about/readme.txt')
data = response.read()
filename = "readme.txt"
file_ = open(filename, 'w')
file_.write(data)
file_.close()

Download image file
Downloading an image from the web works in the same way:

import urllib2
response = urllib2.urlopen('http://upload.wikimedia.org/wikipedia/commons/thumb/f/f8/Python_logo_and_wordmark.svg/260px-Python_logo_and_wordmark.svg.png')
data = response.read()
filename = "image.png"
file_ = open(filename, 'w')
file_.write(data)
file_.close()

Class

Online Courses

Python Class
In Python everything is an object. An object has zero or more methods.
Thus far you have already worked with objects. Let’s take an example:

s = [1,2,3,4]
s.reverse()
s.append(2)
s.remove(1)

In the above example, we have an object named s (a list). This object has the methods reverse(), append() and remove().

Class example

Example 1
To create new types of objects, we must define a class.

class ShoppingList:
    products = []

    def __init__(self):
        print('Shopping list created')

    def add(self, name):
        self.products.append(name)

    def show(self):
        print(self.products)

groceries = ShoppingList()
groceries.add('Peanutbutter')
groceries.add('Milk')
groceries.show()

We create an object named groceries, of the type ShoppingList. We then use the methods add() and show().
We also defined a class named ShoppingList which has these methods defined. In addition there is a method named __init__(), which is always called upon creation of an object.

Example 2
We create an object of the type car, named superCar. The superCar object has one method drive().
In addition to the the __init__ method (also called constructor) that is called when you create new objects.

class Car:
    def __init__(self):
        print('Car created')

    def drive(self):
        print('Engine started')

superCar = Car()
superCar.drive()

Text to speech

We can make the computer speak with Python. Given a text string, it will speak the written words in the English language. This process is called Text To Speech or shortly TTS.

Installation

You need to install one of these two modules: pyttsx or gTTS.
You can simply install with pip:

sudo pip install pyttsx
sudo pip install gTTS

Text to speech

Pyttsx text to speech
Pytsx is a cross-platform text-to-speech wrapper. Underlying it uses different speech engines based on your operating system:

  • nsss – NSSpeechSynthesizer on Mac OS X 10.5 and higher
  • sapi5 – SAPI5 on Windows XP, Windows Vista, and (untested) Windows 7
  • espeak – eSpeak on any distro / platform that can host the shared library (e.g., Ubuntu / Fedora Linux)

Example:

import pyttsx
engine = pyttsx.init()
engine.say('Good morning.')
engine.runAndWait()

gTTS text to speech
gTTS is a module and command line utility to save spoken text to mp3 via the Google Text to Speech (TTS) API. This module supports many languages and sounds very natural.

from gtts import gTTS
import os
tts = gTTS(text='Good morning', lang='en')
tts.save("good.mp3")
os.system("mpg321 good.mp3")

If you want to test it on the command line use:

gtts-cli.py "Hello" -l 'en' -o hello.mp3

Read Gmail using Python

The poplib module can be used to download mails from an email server such as gmail. The protocol used between your computer and the email server is called POP (Post Office Protocol). This protocol can not send mail.

First enable POP support in gmail. .

Connect with gmail
We can conect with gmail using:

import poplib
import string, random
import StringIO, rfc822
import logging

SERVER = "pop.gmail.com"
USER  = "XXXXXX"
PASSWORD = "XXXXXX"

# connect to server
logging.debug('connecting to ' + SERVER)
server = poplib.POP3_SSL(SERVER)
#server = poplib.POP3(SERVER)

# log in
logging.debug('log in')
server.user(USER)
server.pass_(PASSWORD)

Read an email
Read gmail messages using this code:

import poplib
import string, random
import StringIO, rfc822
import logging

SERVER = "pop.gmail.com"
USER  = "XXXXXXXXXXX"
PASSWORD = "XXXXXXXXXXX"

# connect to server
logging.debug('connecting to ' + SERVER)
server = poplib.POP3_SSL(SERVER)
#server = poplib.POP3(SERVER)

# login
logging.debug('logging in')
server.user(USER)
server.pass_(PASSWORD)

# list items on server
logging.debug('listing emails')
resp, items, octets = server.list()

# download the first message in the list
id, size = string.split(items[0])
resp, text, octets = server.retr(id)

# convert list to Message object
text = string.join(text, "\n")
file = StringIO.StringIO(text)
message = rfc822.Message(file)

# output message
print(message['From']),
print(message['Subject']),
print(message['Date']),
#print(message.fp.read())   

We login with the first block. We get all items on the server using server.list(). We go on downloading the first message in the list and finally output it. If you want to output the message data simply uncomment the line message.fp.read()

Get links from webpage

The module urllib2 can be used to download webpage data. Webpage data is always formatted in HTML format. To cope with the HTML format data, we use a Python module named BeautifulSoup.

Get all links from a webpage
To get all links from a webpage:

from BeautifulSoup import BeautifulSoup
import urllib2
import re

html_page = urllib2.urlopen("http://slashdot.org")
soup = BeautifulSoup(html_page)
links = []
for link in soup.findAll('a'):
    links.append(link.get('href'))

print(links)

Explanation
This line downloads the webpage data (which is surrounded by HTML tags):

html_page = urllib2.urlopen("http://slashdot.org")

The next line loads it into a BeautifulSoup object:

soup = BeautifulSoup(html_page)

The link codeblock will then get all links using .findAll(‘a’), where ‘a’ is the indicator for links in html.

links = []
for link in soup.findAll('a'):
    links.append(link.get('href'))

Finally we show the list of links:

print(links)

Socket Server (Network)

A server can be created using sockets. Sockets work on the application layer, it does not specify any protocol and on this level you’d define an application protocol yourself.

Creation of a socket server needs these steps:

  • bind socket to port
  • start listening
  • wait for client
  • receive data

Example socket server

import socket
import sys
 
HOST = ''  
PORT = 7000
 
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print('# Socket created')
 
# Create socket on port
try:
    s.bind((HOST, PORT))
except socket.error as msg:
    print('# Bind failed. ')
    sys.exit()
     
print('# Socket bind complete')
 
# Start listening on socket
s.listen(10)
print('# Socket now listening')
 
# Wait for client
conn, addr = s.accept()
print('# Connected to ' + addr[0] + ':' + str(addr[1]))

# Receive data from client
while True:     
    data = conn.recv(1024)
    line = data.decode('UTF-8')    # convert to string (Python 3 only)
    line = line.replace("\n","")   # remove newline character
    print( line )     
     
s.close()

Once run, a server will be running on localhost port 7000.

# Socket created
# Socket bind complete
# Socket now listening
# Connected to 127.0.0.1:40499

Once running it will wait for messages. To connect with it, use telnet or modify the socket client from the previous section.

Python Socket Client (Network)

Sockets are the basis of any network communication in your computer.  If you open a website, a socket is created in the background. The same thing applies to chat applications or any other network application.

The TCP/IP model
Packets are delivered to another computer using a layered model.  Every layer of the TCP/IP model has a certain purpose. The application layers responsibility is solely the communication language between two applications.  Sockets exist right below the application protocols.

TCP/IP

On the sockets level, you would have to define your own application protocol but you don’t have to worry about network drivers, packet delivery or the internet protocol.

To summarize, sockets work on the application layer.

Python simple socket client
We will create a simple socket client, that mimics a webbrowser. The web uses port 80. The steps a webbrowser does to get a webpage are:

  • create socket
  • get server ip address from domain name
  • connect to server using ip address
  • send request to server
  • receive data (webpage)

In code that will look like:

# Socket client example in python
 
import socket
import sys  

host = 'www.pythonprogramminglanguage.com'
port = 80  # web
 
# create socket
print('# Creating socket')
try:
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
    print('Failed to create socket')
    sys.exit()
     
 
print('# Getting remote IP address') 
try:
    remote_ip = socket.gethostbyname( host )
except socket.gaierror:
    print('Hostname could not be resolved. Exiting')
    sys.exit()
 
# Connect to remote server
print('# Connecting to server, ' + host + ' (' + remote_ip + ')')
s.connect((remote_ip , port))
 
# Send data to remote server
print('# Sending data to server')
request = "GET / HTTP/1.0\r\n\r\n"
 
try:
    s.sendall(request)
except socket.error:
    print 'Send failed'
    sys.exit()
 
# Receive data
print('# Receive data from server')
reply = s.recv(4096)
 
print reply

Of course all data returned will be shown as the raw html (no images, css etc).

 

Statistics

Online Courses

Statistics with Python
Using the math library you can get various statistical values. We start by creating a list of values:

import math

x = [1,2,15,3,6,17,8,16,8,3,10,12,16,12,9]

plot

Five number summary
We can get the five number summary using the math library. The five number summary contains:  minimum, maximum, median, mean and the standard deviation.

import math
import numpy

x = [1,2,15,3,6,17,8,16,8,3,10,12,16,12,9]

print(numpy.min(x))
print(numpy.max(x))
print(numpy.std(x))
print(numpy.mean(x))
print(numpy.median(x))

Boxplot
This code will create a boxplot:

import matplotlib.pyplot as plt
import numpy as np

x = [1,2,15,3,6,17,8,16,8,3,10,12,16,12,9]

plt.boxplot(x)
plt.show()
boxplot
boxplot

Regular expressions

Online Courses

Regular Expressions
Regular Expressions may be used to test if a string (Text) is according to a certain grammar. Sometimes Regular Expressions are called regex or regexp.  We can easily test if a string contains another string:

>>> s = "Are you afraid of ghosts?"
>>> "ghosts" in s
True

You can also test if a string does not contain a substring:

>>> "coffee" not in s
True
>>> 

Grammar
We may define a grammar, to match against any input string. Let’s say we want to match against three digits, we would define a gramamr ‘\d{3}\Z’. Example:

#!/usr/bin/python
import re

s = "123"
matcher = re.match('\d{3}\Z',s)

if matcher:
    print("True")
else:
    print("False")

This will out put “True” if the string s matches the grammar string.

Grammar rules
The permitted grammar for regular expressions is:

\d 	Matches a decimal digit; equivalent to the set [0-9].
\D 	The complement of \d. It matches any non-digit character; equivalent to the set [^0-9].
\s 	Matches any whitespace character; equivalent to [ \t\n\r\f\v].
\S 	The complement of \s. It matches any non-whitespace character; equiv. to [^ \t\n\r\f\v].
\w 	Matches any alphanumeric character; equivalent to [a-zA-Z0-9_]. 
\W 	Matches the complement of \w.
\b 	Matches the empty string, but only at the start or end of a word.
\B 	Matches the empty string, but not at the start or end of a word.
\\ 	Matches a literal backslash.

Logging

Logging is a very powerful way of validating your program executes correctly. In addition you can use it to debug your program.

You can log a process in your program using the module logging.

Online Courses

Example
This message would be shown in the terminal.

import logging
logging.warning('Something went wrong.')

Levels of severity
There are several levels of severity: DEBUG, INFO, WARNING, ERROR, CRITICAL.
You can configure a minimum level of severity, if it’s lower than the set level it’s ignored.

import logging
 
logging.basicConfig(level=logging.WARNING)
logging.debug('Debug message')
logging.error('This is an error')

Another example:

import logging

logging.basicConfig(level=logging.ERROR)
logging.debug('Debug message')
logging.info('Program started..')
logging.info('Loading files')
logging.error('This is an error')

While developing you could set the severity level to logging.DEBUG which will show all messages. This will help you develop faster, find bugs quicker and so on.

Upon release of your program to the market you could set it to debugging.WARNING or debugging.ERROR.

import logging
logging.basicConfig(level=logging.WARNING)
1 2 3