When Batch Processing Makes Sense

Every workload that calls a language model falls somewhere on a spectrum between two extremes. At one end sits a user typing a question into a chat interface and waiting for an answer within a second or two. At the other end sits a system that needs to process ten thousand support tickets, product descriptions, or research documents sometime before tomorrow morning, with nobody watching the screen while it runs. The first case is a real-time workload. The second is a batch workload. Choosing the wrong processing model for a given job is one of the most expensive architectural mistakes you can make when building on the OpenAI SDK, because it affects cost, reliability, code complexity, and how the rest of your system has to be designed.

Real-Time vs. Batch: The Core Distinction

A real-time (synchronous, interactive) request is one where a human or another time-sensitive system is waiting for the response right now. The defining constraint is latency: the user experience degrades if the response takes too long, so you optimize for speed, usually processing one request at a time or a small number in parallel, and you pay standard API pricing per token.

A batch workload is one where the total time to finish all the work matters far more than the time to finish any single item. Nobody is staring at a spinner waiting for item number 4,732 out of 10,000. This shifts the optimization target from per-request latency to overall throughput and cost efficiency.

The OpenAI SDK gives you two fundamentally different tools for these two situations:

Note: Exact Batch API pricing discounts and turnaround windows change over time. Confirm current numbers against the official OpenAI documentation before quoting them to stakeholders or building cost models around them.

Unit 12, Lesson 5 already covered the mechanics of submitting a batch job — building a JSONL file, uploading it, polling for completion, and matching custom_id values back to your original inputs. This lesson does not repeat that. It answers a different, earlier question: how do you decide, before writing any code, whether a given workload should be batch or real-time in the first place?

The Decision Criteria

Four questions reliably separate batch-appropriate workloads from real-time ones.

1. Is there a human waiting synchronously for this specific response?

If a person submitted a form and is looking at a loading spinner, you cannot make them wait an hour. That rules out batch immediately, regardless of volume. Chat assistants, live customer support tools, and autocomplete features are almost always real-time by necessity.

2. Is the volume large enough that per-request overhead matters?

Sending five requests a day gains nothing from batch infrastructure — the engineering cost of building a pipeline, tracking job state, and handling asynchronous results outweighs the savings. Batch processing earns its complexity once you are dealing with hundreds or thousands of items in a single run, where the discounted pricing and the ability to submit everything in one job produce a real difference in cost and operational simplicity.

3. Can the work tolerate delayed results?

If "the summaries will be ready sometime tonight" is an acceptable answer, you have a batch workload. If someone needs the summary of the document they just uploaded before they can continue their task, you don't.

4. Is the work naturally describable as a fixed, enumerable set of independent items?

Batch processing works best when you can enumerate the full list of inputs up front — a table of 50,000 product descriptions to classify, a folder of 10,000 support tickets to tag, an export of a year's worth of transcripts to summarize. It works poorly for open-ended, continuously arriving streams where you would be constantly starting new jobs to catch a handful of new items, because job submission and polling both carry fixed overhead that only pays off at scale.

A Practical Example: Estimating Whether Batch Is Worth It

Suppose you need to classify customer reviews into sentiment categories. Here is a simple way to make the batch-vs-real-time decision data-driven rather than based on gut feeling:

from dataclasses import dataclass


@dataclass
class WorkloadProfile:
    item_count: int
    max_acceptable_delay_hours: float
    is_user_facing_synchronous: bool


def recommend_processing_mode(profile: WorkloadProfile) -> str:
    """Return 'batch' or 'realtime' based on simple, explicit thresholds."""
    if profile.is_user_facing_synchronous:
        return "realtime"

    # Below this volume, batch's fixed overhead (job setup, polling,
    # result reconciliation) isn't worth the discount it buys you.
    MIN_ITEMS_FOR_BATCH = 200

    if profile.item_count < MIN_ITEMS_FOR_BATCH:
        return "realtime"

    if profile.max_acceptable_delay_hours < 1:
        return "realtime"

    return "batch"


# Example usage
nightly_report_job = WorkloadProfile(
    item_count=8000,
    max_acceptable_delay_hours=12,
    is_user_facing_synchronous=False,
)
support_chat_reply = WorkloadProfile(
    item_count=1,
    max_acceptable_delay_hours=0.01,
    is_user_facing_synchronous=True,
)

print(recommend_processing_mode(nightly_report_job))   # "batch"
print(recommend_processing_mode(support_chat_reply))    # "realtime"

This function is intentionally simple — it is not meant to be a sophisticated model, but a concrete, inspectable decision policy. In a real system, MIN_ITEMS_FOR_BATCH should be derived from your own numbers: compare the discounted batch price per token against your standard price, factor in the engineering time to maintain a batch pipeline, and find the volume at which batch actually saves money net of that overhead. The important habit this example demonstrates is turning a vague judgment call ("this feels like it should be a batch job") into an explicit, reviewable policy that a teammate can read and adjust.

Mixed Workloads

Many real systems are not purely one or the other. A common pattern is a hybrid pipeline: the bulk of predictable, high-volume work (nightly re-classification of a product catalog, for instance) runs through the Batch API, while a smaller real-time path handles urgent, user-triggered requests (a merchant needing one product classified immediately after upload). Both paths often share the same prompt templates and parsing logic — only the submission and result-collection mechanism differs. When you design the pipeline in Lesson 2 of this unit, keep this separation in mind: the business logic of building a request and interpreting a response should not need to know whether it will eventually be submitted synchronously or as part of a batch job.

Common Mistakes

• Building a batch pipeline for a workload that's actually small or urgent. Teams sometimes reach for the Batch API because it looks more "production-grade," then discover they've added a 24-hour worst-case turnaround and a polling loop to a workload that only ever has a few dozen items. Match the tool to the actual volume and latency requirement, not to what looks impressive in an architecture diagram.
• Ignoring the batch turnaround window in downstream planning. The Batch API does not guarantee completion within any fixed short window. If a report absolutely must be ready by 8 AM, either submit early enough to absorb the worst case or build a fallback that finishes the remaining items via real-time calls if the batch job is still running past a deadline.
• Treating "large volume" as the only criterion. A workload can be large and still need real-time processing — for example, if it must react to results item-by-item as they arrive (triggering downstream actions per item) rather than being processed as a single reviewable output at the end.

Best Practices

• Write down your decision criteria before building anything. A short, explicit policy like the recommend_processing_mode function above prevents the choice from being re-litigated informally every time a new workload comes up.
• Measure actual per-item cost and turnaround before committing. Run a small pilot batch and a small real-time sample of the same workload, and compare real numbers instead of assuming the discount will be worth the added complexity.
• Design for hybrid from the start when there's any chance you'll need it. Keeping request-building and response-parsing logic independent of the submission mechanism costs little upfront and saves a painful refactor later when a "purely batch" workload grows an urgent real-time requirement.