Python Multiprocessing Example: Process, Pool, and Queue

The multiprocessing module runs Python code in separate operating-system processes, each with its own interpreter and memory. That matches how the official documentation presents it: a threading-like API for process-based parallelism, with Process, pools, queues, and pipes as the building blocks. This guide stays on that path—Process, Pool (map / starmap), Queue, join(), a short look at sharing data, and the errors beginners hit first—not a full survey of every synchronization primitive.

For I/O-bound concurrency inside one process, threads or asyncio are often simpler; for a comparison of threads in one interpreter, see Python multithreading.

Tested on: Python 3.13.3; kernel 6.14.0-37-generic.

Child processes are not reliable inside every in-browser code runner, so every Python fence below uses {run=false}—save the snippet as main.py and run python main.py locally. While updating this page, the examples were executed on Linux with Python 3.13 (same baseline as the Tested-on line) so the patterns behave as shown.

What is multiprocessing in Python?

In the operating system, a process is an isolated execution context: its own memory space, file descriptors, and one or more threads. Python’s multiprocessing module lets you start child processes from your program. The process that calls Process.start() or creates a Pool is the parent; it keeps running while workers execute your target function in parallel. Each child loads a Python interpreter and runs the target you pass to Process or the worker logic inside a Pool. Those children are scheduled by the OS like any other programs, so they can run truly in parallel on multiple CPU cores.

That matters for CPU-bound work in CPython: the Global Interpreter Lock (GIL) prevents more than one thread from executing Python bytecode at once in a single process, so threads help mainly for I/O-bound waiting, not for heavy numeric loops in pure Python. Separate processes each have their own interpreter and their own GIL, so several processes can crunch numbers at the same time.

Processes do not share ordinary Python objects in memory the way threads do. To pass data to a child or get results back, the runtime typically pickles arguments and pipes them through OS primitives (Queue, Pipe, pool channels). That is why worker targets and args must be picklable, and why large or exotic objects are awkward compared to threads. On Windows (and in other spawn configurations), the child also re-imports your main module, which is why every example here wraps process creation in if __name__ == "__main__":—without that guard, you can accidentally start processes recursively at import time.

A Pool is still multiple processes: workers are started once (up to the size you choose), then fed many small tasks through an internal queue until you close the pool—handy when you want to reuse processes instead of paying startup cost for every item.

In short: multiprocessing is process-based parallelism with a familiar API (start, join, queues, pools), trading a bit more overhead and stricter data rules for real multi-core speed on CPU-heavy tasks.

When should you use multiprocessing?

Use it when:

• The task is CPU-bound (heavy computation, large numeric loops, image or data processing) and you have more than one core.
• You can express work as isolated chunks with clear inputs and outputs.

Prefer threads or asyncio when work is I/O-bound (waiting on disk, network, or databases)—extra processes add overhead without much benefit. Prefer a job queue or distributed system when work must span machines; multiprocessing is single-host only.

Python multiprocessing vs threading

Basic multiprocessing example using Process

Process runs a callable in a child process. Start it with start(), then wait with join().

from multiprocessing import Process


def say_hi(name):
    print(f"hello from worker, {name}")


if __name__ == "__main__":
    p = Process(target=say_hi, args=("Ada",))
    p.start()
    p.join()
    print("parent done")

The worker prints from its own process; the parent prints after the child has finished.

Pass arguments to a process

Use the args tuple for positional arguments and kwargs for keyword arguments (both must be picklable).

from multiprocessing import Process


def add(a, b, label="sum"):
    print(label, a + b)


if __name__ == "__main__":
    p = Process(target=add, args=(2, 3), kwargs={"label": "result"})
    p.start()
    p.join()

Process.start() returns None, so keep the Process in a variable and call start() then join() on that object.

Wait for processes using join()

join() blocks the parent until the child terminates. If you pass timeout in seconds, join returns once that much time has passed even if the child is still running—then use is_alive() to see whether it finished. After a normal exit, exitcode is usually 0.

from multiprocessing import Process
import time


def work():
    time.sleep(0.1)


if __name__ == "__main__":
    p = Process(target=work)
    p.start()
    p.join(timeout=2)
    print("alive?", p.is_alive())
    print("exit code", p.exitcode)

A normal exit usually shows exitcode 0; None means still running if join timed out.

Run multiple processes in a loop

Create a list of Process objects, start them all, then join them all so the parent waits for every child.

from multiprocessing import Process


def square(n):
    print(n * n)


if __name__ == "__main__":
    procs = [Process(target=square, args=(i,)) for i in range(3)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()
    print("all workers finished")

Output order can vary because the OS schedules processes independently.

Get results from processes using Queue

A multiprocessing.Queue is process-safe. Workers put results; the parent get them (often after join so producers have finished).

from multiprocessing import Process, Queue


def worker(n, q):
    q.put(n * n)


if __name__ == "__main__":
    q = Queue()
    procs = [Process(target=worker, args=(i, q)) for i in range(4)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()
    results = [q.get() for _ in procs]
    print(sorted(results))

For a single producer and consumer pattern, ensure the consumer does not block forever—either send a sentinel value or join producers first, then drain the queue.

Use multiprocessing Pool

A Pool keeps a fixed number of worker processes and reuses them for many tasks, which is cheaper than spawning a new Process per item for large batches.

from multiprocessing import Pool


def inc(x):
    return x + 1


if __name__ == "__main__":
    with Pool(2) as pool:
        print(pool.map(inc, [1, 2, 3, 4]))

Always use the pool as a context manager (with Pool(...) as pool:) or call close() and join() so workers shut down cleanly.

Pool.map() example

map(func, iterable) splits the iterable across workers and preserves result order (unlike manual processes where completion order can differ).

from multiprocessing import Pool


def is_even(n):
    return n % 2 == 0


if __name__ == "__main__":
    with Pool(3) as pool:
        print(pool.map(is_even, range(6)))

Pool.starmap() example

starmap is like map, but each input must be an iterable of arguments unpacked with *—useful when each task needs several parameters.

from multiprocessing import Pool


def mul(a, b):
    return a * b


if __name__ == "__main__":
    tasks = [(2, 3), (4, 5), (6, 7)]
    with Pool(2) as pool:
        print(pool.starmap(mul, tasks))

Share data between processes

Default variables are not shared: each process gets a copy. To share memory you must use explicit mechanisms from the module.

Use Value and Array

Value and Array allocate shared memory through ctypes types; updates should still be coordinated with locks when multiple writers exist.

from multiprocessing import Process, Value, Lock


def bump(counter, lock):
    for _ in range(1000):
        with lock:
            counter.value += 1


if __name__ == "__main__":
    lock = Lock()
    counter = Value("i", 0)
    procs = [Process(target=bump, args=(counter, lock)) for _ in range(4)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()
    print(counter.value)

You should see 4000 when each of four workers increments one thousand times under the lock.

Use Manager list and dict

A Manager runs a server process that holds Python objects proxied to workers—flexible but slower than Value/Array.

from multiprocessing import Process, Manager


def append_item(shared_list, value):
    shared_list.append(value)


if __name__ == "__main__":
    with Manager() as manager:
        shared = manager.list()
        procs = [Process(target=append_item, args=(shared, i)) for i in range(3)]
        for p in procs:
            p.start()
        for p in procs:
            p.join()
        print(list(shared))

Handle exceptions and exit codes

If a child process raises an uncaught exception, Process.exitcode is typically non-zero (1 on many platforms); the traceback prints to the child’s stderr. Wrap risky work in the target function and use a Queue to ship structured errors back to the parent if you need to handle them programmatically.

from multiprocessing import Process


def bad():
    raise RuntimeError("worker failed")


if __name__ == "__main__":
    p = Process(target=bad)
    p.start()
    p.join()
    print("exitcode", p.exitcode)

Common multiprocessing errors

Missing if name == "main"

On Windows (and whenever the start method is spawn), the child imports your module again. Any process-creation code at module level runs in every child and can recurse or fail. Put process and pool creation inside if __name__ == "__main__": so it runs only in the real entry process:

from multiprocessing import Process


def task():
    print("ok")


if __name__ == "__main__":
    p = Process(target=task)
    p.start()
    p.join()

Can't pickle local object

The target callable and args must be picklable. Nested functions, lambdas used as targets, and many bound methods break pickling. Define workers at module top level (or use a picklable functools.partial of a top-level function).

• Queue hangs or process never exits

Common causes: a parent blocked on Queue.get while children never put; a child waiting on a full queue with no reader; or forgetting to join producers before assuming the queue is complete. Join writers, use timeouts while debugging, and consider a sentinel None to mark the end of a stream of results. For logging from workers, see Python logging.

Multiprocessing best practices

• Keep the if __name__ == "__main__": guard around all process and pool creation on portable code.
• Prefer small picklable messages over huge objects on Queue.
• Size Pool to your CPU (os.cpu_count()), leaving headroom for the OS; more processes than cores can slow CPU-bound work.
• Close pools and join processes so interpreters do not leak zombie workers.
• Measure before scaling; for wall-clock timing patterns, see Python measure execution time.

Python multiprocessing quick reference table

Summary

Python’s multiprocessing module gives you process-based parallelism: Process for explicit workers, join() to synchronize, and Queue to pass results back safely. For batch CPU work over a function and an iterable, Pool.map and Pool.starmap are the usual tools. Sharing mutable state needs Value, Array, or a Manager, always with a clear story about locking and lifetime. Most beginner failures come from missing the __main__ guard, unpicklable targets, or queue deadlocks—fix those first, then profile and tune pool size on real hardware.

References

• multiprocessing — Process-based parallelism
• Logging recipes for multiprocessing (when you outgrow print debugging)