Why Waiting Synchronously Wastes Time

Every example in this course so far has made one request, waited for it to finish, and only then moved on to the next line of code — perfectly fine for a script processing one thing at a time, but a real constraint the moment an application needs to make many independent requests around the same time. A synchronous call to client.responses.create() spends most of its time simply waiting for a network round trip to complete, during which the program does nothing else at all; processing a hundred independent documents one after another, each waiting on its own network round trip in turn, takes roughly a hundred times as long as it would if those waits could overlap. Async support exists specifically to let those waits overlap.

The Async Client

The SDK provides an async counterpart to the client used throughout this course, whose methods are awaited rather than called directly, for use inside an async def function.

import asyncio
from openai import AsyncOpenAI

async_client = AsyncOpenAI()

async def get_response(prompt: str):
    response = await async_client.responses.create(
        model="gpt-5.6-terra",
        input=prompt,
    )
    return response.output_text

result = asyncio.run(get_response("Summarize the plot of a short mystery story."))
print(result)

Note: The exact async client class name and whether every synchronous method has a directly corresponding async method can vary by SDK version. Confirm the current async interface against your installed SDK version's documentation.

Used this way — one call, awaited, inside asyncio.run() — the async client behaves identically to the synchronous version and offers no advantage on its own; the benefit only appears once multiple independent calls are allowed to run concurrently instead of one after another, which is what the rest of this lesson builds toward.

Running Multiple Requests Concurrently

asyncio.gather() runs several async calls concurrently, waiting only as long as the slowest one takes, rather than for the sum of all of them.

async def summarize_document(document_text: str) -> str:
    response = await async_client.responses.create(
        model="gpt-5.6-terra",
        input=f"Summarize this document in two sentences:\n\n{document_text}",
    )
    return response.output_text

async def summarize_all(documents: list) -> list:
    tasks = [summarize_document(doc) for doc in documents]
    return await asyncio.gather(*tasks)

documents = ["Document one's full text...", "Document two's full text...", "Document three's full text..."]
summaries = asyncio.run(summarize_all(documents))
for summary in summaries:
    print(summary)

Summarizing three documents this way takes roughly as long as summarizing the single slowest one, rather than the sum of all three — for a batch of a hundred documents needing individual, immediate responses (as opposed to the non-time-sensitive bulk work Lesson 5's Batch API is built for), this difference is the difference between a task that finishes in seconds and one that takes minutes.

Bounding Concurrency Rather Than Running Everything at Once

Running every request in an unbounded asyncio.gather() call risks launching far more concurrent requests than the account's actual rate limit (Lesson 3) can sustain — a hundred documents launched all at once might comfortably fit under a rate limit, but ten thousand almost certainly won't. A semaphore caps how many requests actually run concurrently, letting the rest wait their turn.

async def summarize_with_limit(document_text: str, semaphore: asyncio.Semaphore) -> str:
    async with semaphore:
        response = await async_client.responses.create(
            model="gpt-5.6-terra",
            input=f"Summarize this document in two sentences:\n\n{document_text}",
        )
        return response.output_text

async def summarize_all_bounded(documents: list, max_concurrent: int = 10) -> list:
    semaphore = asyncio.Semaphore(max_concurrent)
    tasks = [summarize_with_limit(doc, semaphore) for doc in documents]
    return await asyncio.gather(*tasks)

summaries = asyncio.run(summarize_all_bounded(documents, max_concurrent=10))

asyncio.Semaphore(10) allows at most 10 of the scheduled tasks to actually be inside the async with semaphore: block — making an actual request — at any one time; the rest wait until a slot frees up. Choosing an appropriate max_concurrent value connects directly back to Lesson 3's rate-limiting discussion: too high a value and the account's actual rate limit gets exceeded regardless of how careful the async code is; too low and concurrency's benefit is left mostly unused.

Handling Failures Within a Concurrent Batch

By default, asyncio.gather() raises the first exception it encounters and cancels the rest of the still-running tasks — often not the desired behavior when some documents succeeding and others failing independently is an acceptable, even expected, outcome.

async def summarize_all_with_error_handling(documents: list, max_concurrent: int = 10) -> list:
    semaphore = asyncio.Semaphore(max_concurrent)
    tasks = [summarize_with_limit(doc, semaphore) for doc in documents]
    results = await asyncio.gather(*tasks, return_exceptions=True)

    successes, failures = [], []
    for document, result in zip(documents, results):
        if isinstance(result, Exception):
            failures.append({"document": document, "error": str(result)})
        else:
            successes.append(result)

    return successes, failures

successes, failures = asyncio.run(summarize_all_with_error_handling(documents))
print(f"{len(successes)} succeeded, {len(failures)} failed")

return_exceptions=True changes gather()'s behavior so a failed task's exception is returned as a value in the results list instead of being raised immediately and cancelling every other still-running task — this is almost always the more appropriate choice for a batch of otherwise-independent requests, since one document's failure (perhaps due to a malformed input triggering the 400 error Lesson 1 covered) has no bearing on whether the other ninety-nine should also be abandoned.

When Async Is Worth the Added Complexity

Async code is genuinely more complex to write and reason about than the synchronous code used throughout this course, and it isn't the right choice for everything.

The web-server case deserves particular attention going into Unit 14: a synchronous, blocking call inside a request handler ties up that handler for the entire duration of the wait, unable to serve any other incoming request in the meantime — a web application serving multiple concurrent users needs an async client specifically to avoid one slow request blocking every other user's request from being handled at all.

Common Mistakes

Mixing synchronous and async client calls inside the same async function, calling the blocking client.responses.create() instead of await async_client.responses.create() inside an async def, which defeats the entire purpose of using async code in the first place.

Launching unbounded concurrency with asyncio.gather() over a large list of tasks, risking exceeding the account's actual rate limit by launching far more simultaneous requests than it can sustain.

Letting one failed task cancel an entire batch of otherwise-independent concurrent requests, by omitting return_exceptions=True when partial success is an acceptable, expected outcome.

Using async for a simple script with no concurrency need, adding meaningful code complexity for a benefit that only exists when multiple independent operations can actually run at once.

Best Practices

Use the async client and asyncio.gather() for any batch of many independent, time-sensitive requests, rather than looping through them synchronously one at a time.

Bound concurrency with a semaphore sized to stay comfortably under the account's actual rate limit, connecting directly to Lesson 3's rate-limiting guidance.

Use return_exceptions=True when partial failure within a batch is acceptable, and handle each result explicitly rather than letting one failure cancel the rest.

Reserve async specifically for genuine concurrency needs — many simultaneous requests, or a web server handling multiple users — rather than adding it to code that only ever does one thing at a time.