Concurrency, Async & the GIL

“Make it faster” usually means one of two very different things: the program is waiting (on the network, disk, or a model API) or it is computing (crunching numbers). Python gives you different tools for each, and picking the wrong one is a classic mistake. The key that unlocks the whole topic is the GIL.

The GIL, briefly

CPython has a Global Interpreter Lock — only one thread executes Python bytecode at a time. The consequence is stark:

• For CPU-bound work (parsing, math, image processing), threads do not give you real parallelism. You need multiple processes.
• For I/O-bound work (HTTP calls, DB queries, LLM requests), threads and async work beautifully, because a thread waiting on the network releases the GIL and lets others run.

Most AI-engineering workloads are I/O-bound — you spend your time waiting on model APIs — which makes async the tool you’ll reach for most.

Three models of concurrency

asyncio: doing nothing, efficiently

asyncio runs many I/O operations concurrently on a single thread. While one coroutine is awaiting a response, the event loop runs another. There are no threads to manage and no locks for shared state — but the rule is strict: never call a blocking function inside async code, or you freeze the entire loop.

import asyncio, httpx

async def fetch(client, url):
    r = await client.get(url)          # yields control while waiting
    return r.json()

async def main(urls):
    async with httpx.AsyncClient() as client:
        # fire all requests concurrently, gather when all complete
        return await asyncio.gather(*(fetch(client, u) for u in urls))

results = asyncio.run(main(urls))

Calling ten APIs sequentially takes the sum of their latencies; gather-ing them takes the max. When you’re fanning out to embedding and LLM endpoints, this is the difference between a snappy request and a sluggish one.

multiprocessing: real parallelism for real computation

When the work is genuinely CPU-bound, sidestep the GIL with processes:

from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor() as pool:
    results = list(pool.map(expensive_pure_function, chunks))

Each process has its own interpreter and GIL, so they run truly in parallel. The cost is that data must be serialized to cross the process boundary, so it pays off only when each task does substantial work.

A decision checklist

1. Is the bottleneck waiting or computing? Profile before you guess.
2. Waiting → asyncio (or threads if you must use a blocking library).
3. Computing → multiprocessing, or push the hot loop into a vectorized library like NumPy that releases the GIL under the hood.
4. Neither is enough? The fastest code is the code you don’t run — cache results, batch requests, and remove redundant work first.

Interview-ready summary: “Threads for I/O, processes for CPU, async for lots of I/O — because the GIL means Python threads don’t parallelize computation.”

When to reach for this

Variations

Try it

These exercises are about choosing the right execution model, not sprinkling async everywhere.

async def fetch_all(client, urls):
    results = []
    for url in urls:
        results.append(await fetch_json(client, url))
    return results

import asyncio


async def fetch_all(client, urls):
    tasks = [fetch_json(client, url) for url in urls]
    return await asyncio.gather(*tasks)

import asyncio
from concurrent.futures import ProcessPoolExecutor


def score_document(doc: str) -> float:
    return expensive_python_scoring(doc)


async def score_documents(docs: list[str]) -> list[float]:
    loop = asyncio.get_running_loop()
    with ProcessPoolExecutor() as pool:
        tasks = [loop.run_in_executor(pool, score_document, doc) for doc in docs]
        return await asyncio.gather(*tasks)

from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor


def call_vendor(prompt: str) -> dict:
    return blocking_sdk.generate(prompt)


def hash_image(path: str) -> str:
    return pure_python_image_hash(path)


with ThreadPoolExecutor(max_workers=20) as threads:
    responses = list(threads.map(call_vendor, prompts))

with ProcessPoolExecutor() as processes:
    hashes = list(processes.map(hash_image, image_paths))