When a Request Doesn't Need an Immediate Answer

Every example so far in this course has assumed a request needs its response right away — a user is waiting, an agent's next step depends on the current one finishing, a web request holds a connection open until a reply arrives. A meaningful category of real work doesn't share that constraint at all: classifying ten thousand support tickets overnight, generating embeddings (Unit 10) for an entire document archive, summarizing a week's worth of accumulated logs before a morning report — none of these need a response within seconds, and all of them involve far more individual requests than would be practical to send one at a time through the synchronous APIs used throughout this course. The Batch API is built specifically for this shape of work: large volume, no immediate latency requirement, submitted together and collected later.

What Batch Processing Trades For What

Submitting work as a batch means accepting a longer, less predictable turnaround time — typically up to 24 hours — in exchange for two things a synchronous request doesn't offer: a meaningfully lower per-token cost, and freedom from needing to manage rate limiting (Lesson 3) across thousands of individual synchronous calls yourself.

Note: The exact batch completion window, the specific cost discount compared to synchronous requests, and which endpoints support batch submission can all change over time. Confirm current batch pricing and turnaround guarantees against the current official documentation before relying on a specific number for capacity planning.

This trade only makes sense when the underlying work genuinely doesn't need an immediate answer — the batch API is not a way to make an interactive chat application or one of Unit 11's agents faster or cheaper; it specifically fits work that was already going to run as a background job.

Building a Batch Input File

A batch request is submitted as a file containing one JSON object per line (JSONL), where each line specifies its own custom_id, the endpoint to call, and the request body — structurally similar to a loop of individual client.responses.create() calls, just described up front instead of executed one at a time.

import json

support_tickets = [
    {"id": "ticket-001", "text": "The app crashes every time I try to log in."},
    {"id": "ticket-002", "text": "I was charged twice for my subscription this month."},
    {"id": "ticket-003", "text": "How do I export my data to CSV?"},
]

def build_batch_input_file(tickets: list, output_path: str) -> None:
    with open(output_path, "w") as f:
        for ticket in tickets:
            request_line = {
                "custom_id": ticket["id"],
                "method": "POST",
                "url": "/v1/responses",
                "body": {
                    "model": "gpt-5.6-terra",
                    "input": f"Classify this support ticket into one category "
                             f"(billing, technical, general): {ticket['text']}",
                },
            }
            f.write(json.dumps(request_line) + "\n")

build_batch_input_file(support_tickets, "batch_input.jsonl")

custom_id is what makes it possible to match each result back to the ticket that produced it once the batch completes — since batch results can arrive in a different order than they were submitted, or with some entries failing while others succeed, relying on list position to match a result to its original input is unreliable in a way that relying on custom_id is not.

Submitting the Batch

Submitting a batch is a two-step process: first uploading the JSONL file, then creating a batch job that references the uploaded file.

from openai import OpenAI

client = OpenAI()

uploaded_file = client.files.create(
    file=open("batch_input.jsonl", "rb"),
    purpose="batch",
)

batch_job = client.batches.create(
    input_file_id=uploaded_file.id,
    endpoint="/v1/responses",
    completion_window="24h",
)

print(f"Batch job submitted: {batch_job.id}, status: {batch_job.status}")

Note: The exact method names, required parameters (such as the purpose value and available completion_window options), and the batch job lifecycle's status values can vary by SDK version. Confirm the current batch submission interface against your installed SDK version's documentation.

The endpoint parameter tells the platform which API every line in the input file is targeting — every request within a single batch file needs to target the same endpoint, so a batch mixing embedding requests (Unit 10) and response requests would need to be split into two separate batch files rather than combined into one.

Polling for Completion

A batch job doesn't complete immediately — checking its status periodically is how an application finds out when results are ready to retrieve.

import time

def wait_for_batch_completion(client, batch_id: str, poll_interval_seconds: int = 60):
    while True:
        batch_job = client.batches.retrieve(batch_id)
        if batch_job.status in ("completed", "failed", "expired", "cancelled"):
            return batch_job
        print(f"Batch status: {batch_job.status}, checking again in {poll_interval_seconds}s")
        time.sleep(poll_interval_seconds)

completed_batch = wait_for_batch_completion(client, batch_job.id)
print(f"Final status: {completed_batch.status}")

Polling at a reasonable interval — once a minute is typical for a job with an hours-long completion window — rather than in a tight loop avoids wasting requests checking a status that's unlikely to have changed in the last second; for a batch expected to take hours, a much longer poll interval (or a scheduled check rather than a blocking wait at all) is usually more appropriate than the tight polling loops this course has used elsewhere for fast, synchronous operations.

Retrieving and Matching Results

Once a batch completes, its results are available as an output file (and, separately, an error file for any individual requests that failed), matched back to the original inputs through custom_id.

def process_batch_results(client, completed_batch) -> dict:
    output_file_content = client.files.content(completed_batch.output_file_id)
    results_by_ticket_id = {}

    for line in output_file_content.text.strip().split("\n"):
        result = json.loads(line)
        ticket_id = result["custom_id"]
        response_body = result["response"]["body"]
        results_by_ticket_id[ticket_id] = response_body["output"][0]["content"][0]["text"]

    return results_by_ticket_id

results = process_batch_results(client, completed_batch)
for ticket_id, classification in results.items():
    print(f"{ticket_id}: {classification}")

Note: The exact shape of each result line (including where the actual model output is nested within response.body) can vary by SDK version and by which endpoint the batch targeted. Confirm the current output file format against your installed SDK version's documentation before parsing it in production code.

Building the results_by_ticket_id dictionary keyed on custom_id — rather than assuming results appear in the same order the original tickets were submitted — is what makes this matching reliable regardless of how the platform orders or groups the completed results internally.

Handling Partial Failures Within a Batch

Not every individual request within a batch necessarily succeeds — a batch can complete overall while some of its individual line items failed, and these show up in a separate error file rather than silently disappearing.

def check_for_batch_errors(client, completed_batch) -> list:
    if not completed_batch.error_file_id:
        return []

    error_file_content = client.files.content(completed_batch.error_file_id)
    failed_ticket_ids = []
    for line in error_file_content.text.strip().split("\n"):
        error_entry = json.loads(line)
        failed_ticket_ids.append(error_entry["custom_id"])
    return failed_ticket_ids

failed_ids = check_for_batch_errors(client, completed_batch)
if failed_ids:
    print(f"{len(failed_ids)} requests failed and may need to be resubmitted: {failed_ids}")

Checking for an error file explicitly, rather than assuming a completed batch status means every individual request within it succeeded, is what catches the case where 9,997 of 10,000 tickets classified successfully and three failed for reasons (a malformed input, an individual request exceeding a token limit) worth investigating and potentially resubmitting on their own.

Common Mistakes

Using the Batch API for work that actually needs an immediate response, applying a mechanism built for non-time-sensitive bulk work to an interactive use case it was never designed for.

Relying on result ordering instead of custom_id to match results back to inputs, producing silently mismatched results if the platform returns entries in a different order than submitted.

Assuming a completed batch status means every individual request succeeded, missing partial failures that only appear in a separate error file.

Polling for batch completion in a tight loop, wasting requests checking a status that, for a job with a multi-hour completion window, is very unlikely to have changed within the last few seconds.

Best Practices

Reserve the Batch API for genuinely non-time-sensitive, high-volume work, where the longer turnaround time and lower cost are an acceptable and beneficial trade.

Assign a meaningful, unique custom_id to every request in a batch, and use it — not list position — to match results back to their original inputs.

Always check for and handle a batch's error file, rather than assuming a successfully completed batch job means every individual request within it succeeded.

Poll for batch completion at an interval proportional to the expected completion window, rather than checking in a tight loop suited to fast, synchronous operations.