Threads in Python

A thread is a sequence of instructions that runs in parallel with other threads inside a single process. Every process starts at least one thread — the main one. A process can create as many threads as the operating system settings allow, or until it runs out of RAM.

When a process starts, the operating system allocates a region of memory and other resources for it, and the main thread uses them. If a single thread is running in the process, only one operation is executed at any given moment. If there are several threads, they can run their operations in parallel, but the number of simultaneous operations cannot exceed the number of cores in the processor.

Each thread is given its own stack — a region of RAM — and threads also share access to the common memory of the process. For example, if there is a list of data in shared memory, it can be read and modified from threads created within the process.

In Python, working with threads is done using the threading module.

Creating a thread

Let's look at the simplest example of creating a thread.

PythonCopy

import time
from threading import Thread, current_thread


def foo(text):
    time.sleep(2)
    print('[foo] Current thread', current_thread().name)
    print('msg:', text)


t = Thread(target=foo, args=('Hello world',))
t.start()

print('[main] Current thread', current_thread().name)
print('Waiting for thread end...')

t.join()

print('Threads finished')

Example of the execution output in the console:

TerminalCopy

[main] Current thread MainThread
Waiting for thread end...
[foo] Current thread foo-thread
msg: Hello world
Threads finished

Let's look at the example in more detail. First we declare the function foo, which will run in a separate thread. Then, using the Thread class, we create a thread instance, passing a callable object to the target parameter (in our case the foo function) and a tuple of the function's arguments to the args parameter. The name parameter lets you give the thread a meaningful name. The start() method starts the thread. The join() method lets you wait for the program's threads to finish.

The threading module also has a current_thread() function, which returns the instance of the current thread associated with the execution context of the callable object. By adding the thread name to the output, you can see which threads the code runs in.

Threads consume fewer resources and can also work with any type of data when passing parameters, without requiring serialization, unlike processes.

This is how a thread works: the function runs independently of the main script code, or vice versa.

Daemon threads

If you don't add the t.join() statement to the main program's code, the main code will finish, but if some threads keep working, the program's overall execution will complete only after all threads have finished.

This behavior is not always desirable, since threads that perform background tasks (for example, mounting devices) may run continuously. To tell the main program that a process is a background one and there is no need to wait for it to finish, add the daemon=True parameter when creating the thread.

PythonCopy

t = Thread(target=foo, args=('Hello world',), daemon=True)

In this case the thread's execution stops immediately when the main program finishes, without waiting for it to complete.

The ThreadPoolExecutor class

Often you'll need to start several threads at once — for example, to process a list of several files or requests to several URLs. The concurrent.futures library provides the ThreadPoolExecutor class to simplify handling parallel threads.

A ThreadPoolExecutor instance is usually created with the with context manager in order to define the executable blocks and clean up the threads after execution.

The class's main methods are submit and map.

submit lets you create a thread for a single function with the parameters passed to it. Syntax: submit(func: Callable, [*args, **kwargs]).

map lets you start several threads for the specified callable object and an array of input parameters for it. Syntax: map(func, *iterables, timeout=None, [chunksize]).

Let's look at an example:

PythonCopy

from concurrent.futures import ThreadPoolExecutor
import time
from threading import current_thread


def sum_len(data: list[int]) -> int:
    return sum(data)


def foo(text: str) -> int:
    time.sleep(len(text))
    print('[foo] Current thread:', current_thread().name)
    print('msg:', text)
    return len(text)


fruit_list = ['apple', 'banana', 'pineapple']

with ThreadPoolExecutor(max_workers=3) as pool:
    print('[main] Current thread:', current_thread().name)
    results = pool.map(foo, fruit_list)
    print('results type:', type(results))

    result = pool.submit(sum_len, results)
    print('result type:', type(result))

    print('Waiting for thread end...')

print('result:', result.result())
print('Threads finished')

Console output:

TerminalCopy

[main] Current thread: MainThread
results type: <class 'generator'>
result type: <class 'concurrent.futures._base.Future'>
Waiting for thread end...
[foo] Current thread: ThreadPoolExecutor-0_0
msg: apple
[foo] Current thread: ThreadPoolExecutor-0_1
msg: banana
[foo] Current thread: ThreadPoolExecutor-0_2
msg: pineapple
result: 20
Threads finished

This code gets the length of the string values in a list and then finds their sum. First we do the necessary imports and define two functions: foo, which returns the length of a string, and sum_len, which returns the sum of the lengths of a list's elements. We define the fruit_list list for testing.

Then, using the with context manager, we create a thread pool instance. As a parameter we specify the maximum number of workers via max_workers.

With the pool.map statement we specify that we need to run the foo function for each element of the fruit_list list in a separate thread and write the result to the results variable. Note the type of the results variable — it is a generator.

The pool.submit statement indicates that we should apply the sum_len function to the result of the previous threads and write the obtained value to the result variable. The type of the result variable will be concurrent.futures._base.Future (also called a 'future'). A future is an awaitable object. To get its computed result, you can use its .result() method.

Conclusion

Let's once again go over the main advantages and disadvantages of threads in Python.

Advantages:

Shared memory between threads is available.

Well suited for parallelizing IO operations (input/output operations).

More lightweight, requiring less memory and fewer resources.

Disadvantages:

There is no way to interrupt a thread's execution programmatically.

The GIL prevents threads from running simultaneously.