Threading Introduction for Python

Threading lets a Python program do two things at the same time. A download can run while the interface stays responsive, for example.

Scripting is where Python really pays off. The Python Scripting Essentials Bundle has the kind of small automation projects you would build for work.

In this article we look at the threading module, how to start a thread and what the GIL means for your code.

With the threading module you start threads from your main program. This page looks at how that works and what the GIL means for it.

Note: Although threading might sound like parallel execution, it’s vital to remember that it isn’t genuinely parallel. The operating system continues to execute threads in a sequential manner.

Understanding Threads

At its core, a thread is an individual stream or flow of execution. So, when you’re employing threads, it seems like your program is executing multiple tasks simultaneously. But they aren’t running exactly concurrently.

In today's tech landscape, many processors possess multiple cores. You can dedicate a separate process to each core. While they operate on individual cores, these processes aren’t running simultaneously.

For using threading in Python, one must import the threading module. Here’s how you can craft a thread:

 
my_thread = threading.Thread(target=thread_function, args=(1,))

The thread in the above example will initiate the execution of a function dubbed thread_function(). A sample function can be:

def thread_function(name):
    logging.info("Thread %s: initiated", name)
    time.sleep(2)
    logging.info("Thread %s: concluded", name)

For those situations where a thread needs to be operational over prolonged periods, you can either assign it a specific duration or encapsulate it within a loop.

Threads won’t commence by themselves. The above-mentioned code merely sets up the thread. To kick off a thread, use the .start() method:

 
my_thread.start()

Below is a Python snippet, demonstrating how a thread gets activated from the primary program. It uses the logging module to display relevant data:

import logging
import threading
import time

def thread_function(name):
    logging.info("Thread %s: initiated", name)
    time.sleep(2)
    logging.info("Thread %s: concluded", name)

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")
    logging.info("Main program, prior to thread creation")
    my_thread = threading.Thread(target=thread_function, args=(1,))
    logging.info("Main program, prior to thread initiation")
    my_thread.start()
    logging.info("Main program, awaiting thread's conclusion")
    logging.info("Concluding thread and main program")

Managing Multiple Threads

While the sample code provided only integrates two threads (the newly created one and the main thread), Python enables the initiation of numerous threads within a program.

With multi-threaded programs, multiple processes transpire concurrently, with every thread contributing towards the end-goal.

The process to commence multiple threads remains identical, as shown above:

 
my_thread = threading.Thread(target=thread_function, args=(1,))

However, the introduction of .join() ensures that the program waits for all threads to culminate before it terminates.

 
my_thread.join()

The join() function instructs a thread to linger until the other threads wrap up:

import logging
import threading
import time

def thread_function(name):
    logging.info("Thread %s: initiated", name)
    time.sleep(2)
    logging.info("Thread %s: concluded", name)

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")
    logging.info("Main program, prior to thread creation")
    
    threads = []
    for index in range(5):
        logging.info("Thread created")
        my_thread = threading.Thread(target=thread_function, args=(1,))
        threads.append(my_thread)
        my_thread.start()
    
    for index, thread in enumerate(threads):
        thread.join()
        logging.info("Thread %d concluded", index)

    logging.info("Main program, awaiting thread's conclusion")

Threads and Queues

Using the modules threading and queue, you can initiate a set of threads. Start by crafting a queue using the Queue() function. Following this, a set of URLs is employed as parameters during thread construction.

Threads get shaped in the loop via the threading.Thread() call. For these threads to commence, you must invoke the thread.start() method. Every thread then proceeds to execute the getUrl method:

 
import queue
import threading
import urllib.request

def getUrl(q, url):
    print('getUrl('+url+') initiated by a thread.')
    q.put(urllib.request.urlopen(url).read())

theurls = ["http://google.com", "http://google.de","http://google.ca"]

threadQueue = queue.Queue()

for u in theurls:
    t = threading.Thread(target=getUrl, args=(threadQueue,u))
    t.daemon = True
    t.start()

output = threadQueue.get()

To visualize this, consider your main program igniting several new threads. Each of these threads runs a distinct task “concurrently”.

In essence, a Python program can incorporate an indefinite number of threads. Each thread operates on specific data, contributing towards the program’s overall objective.

The fastest way to learn this is to write it yourself. The PyChallenge exercises run in your browser, no install needed.