Python: concurrent.futures and multiprocessing

Description

The previous skill showed that threads in standard Python can't compute in parallel: only one of them runs Python code at a time. When work is CPU-heavy - scoring, parsing, image resizing - the usual answer is processes: several separate Python interpreters, each with its own GIL, running on different cores. Python's concurrent.futures module gives threads and processes the same simple interface, so switching between them is often a one-word change.

That one word changes a lot, though. A team moved a CPU-heavy scoring job from threads to processes and hit three walls in a row:

1. PicklingError: Can't pickle <function <lambda> ...> - the scoring function was a lambda.
2. Once that was fixed, a results cache the workers filled stayed empty in the main program.
3. Running the function over 50,000 small items was slower than a plain loop.

All three come from one fact: a process is a separate program with its own memory, and everything sent between processes has to be packed up and unpacked. By the end of this lesson you'll be able to fix each one.

We'll go from simple to more technical:

1. concurrent.futures: running tasks in a pool and collecting results.
2. Threads or processes - which pool when.
3. How worker processes start, and why the __main__ guard matters.
4. What can be sent to a process, and what never comes back.
5. Making process pools fast.

Running tasks in a pool

A pool keeps a few workers ready. You give it tasks; it gives you back futures - placeholders for results that aren't ready yet:

from concurrent.futures import ThreadPoolExecutor, as_completed

with ThreadPoolExecutor(max_workers=8) as pool:   # the with block waits for every task at the end
    future = pool.submit(fetch, url)              # start one task, get a Future
    page = future.result(timeout=5)               # wait for its value - or its exception

    for result in pool.map(fetch, urls):          # many tasks, results in input order
        handle(result)

    futures = {pool.submit(fetch, u): u for u in urls}
    for f in as_completed(futures):               # results in the order they finish
        handle(futures[f], f.result())

What to know about each:

• result() raises the task's own exception - the original ValueError, not a wrapper (Java wraps it in ExecutionException). If nobody ever calls result(), a failed task fails silently.
• map returns results in input order, however long each one takes: tasks sleeping 0.3, 0.1 and 0.2 s came back as [0.3, 0.1, 0.2]. When it reaches a task that failed, it raises that exception and you lose the results after it (a test got [10], then ValueError: bad input 2).
• as_completed gives futures back as they finish - [0.1, 0.2, 0.3] for the same tasks - so each failure can be handled on its own.
• Stopping early: leaving the with block waits for everything. pool.shutdown(cancel_futures=True) (Python 3.9+) cancels every task that hasn't started yet - in a test, 4 of 5 queued tasks.

Threads or processes?

Both pools have the same interface; pick by what the work spends its time on:

Measured on an 8-core laptop: four CPU-bound tasks took 0.47 s one after another, 0.50 s on four threads, and 0.14 s on four processes.

How worker processes start

ProcessPoolExecutor uses the multiprocessing module, which can start a worker in three ways:

With spawn and forkserver, each worker imports your main file, so its top-level code runs again in every worker. A pool created at the top level of a script would try to start more pools inside the workers. That's why every script that uses processes needs the guard:

def score(item):
    ...

if __name__ == "__main__":                       # runs in the parent only
    with ProcessPoolExecutor() as pool:
        results = list(pool.map(score, items))

fork avoids the re-import, but copying a running program has its own danger: if another thread was holding a lock at that moment, the copy holds it forever. That's why Python 3.14 stopped using fork by default on Linux. Code that quietly relied on fork - workers seeing variables the parent set after startup - behaves differently on 3.14: pass what workers need as arguments.

What crosses the boundary

Workers don't share memory with the parent. The function, its arguments and its return value are pickled - turned into bytes, sent through a pipe, and rebuilt on the other side:

• Only things that can be found by name can be sent. Functions and classes defined at the top level of a module work. Lambdas and functions defined inside other functions don't - hence PicklingError: Can't pickle <function <lambda>. Move the function to module level.
• Changes in a worker stay in the worker. A worker that adds to a module-level list or dict changes its own copy: after three workers appended to a list, the parent's list was still []. Return results from the function instead. If you really need shared state, use a database, multiprocessing.Manager, or multiprocessing.shared_memory.

Making process pools fast

Every task sent to a process costs time: pickling, a write to a pipe, waking the worker, and the same on the way back. For tiny tasks, that overhead is bigger than the work. Mapping a trivial function over 50,000 items on a four-process pool took about 3.5 s with the default of one item per message, and 0.02 s with chunksize=1000:

results = list(pool.map(score, items, chunksize=1000))   # send items in batches

Rules of thumb:

• Send work in batches big enough that each takes at least a few milliseconds.
• Send little data in and out: pass a file name or an id, not a huge object, and return small results.
• Create the pool once and reuse it - starting processes takes tens of milliseconds each.

Common mistakes

Check yourself

Try to answer before opening each one. Recalling beats re-reading.

1. Why can't you submit a lambda to a ProcessPoolExecutor?

2. pool.map(f, items) fails on the second item. What do you get?

3. Why is ProcessPoolExecutor.map over 50,000 tiny items slow, and what fixes it?

Key takeaways

• Threads for waiting, processes for computing in Python - with the same concurrent.futures interface.
• result() re-raises the task's exception; map keeps input order and stops at the first failure; as_completed gives finishing order.
• Workers import your main file (spawn, and forkserver on Linux since 3.14): guard it with __main__.
• Only module-level functions can be sent; changes in workers don't come back - return results.
• Batch small tasks with chunksize, and send little data each way.