Compare Batches and Requests

Two Ways to Call the Same Model

Every answer in this course has come from the same models, whether it was requested synchronously or through a batch. What differs is the arrangement: a synchronous request is answered while you wait, and a batch is answered later, in bulk, at half the price. Choosing between them is therefore a question about trade-offs, not about which one is correct. This lesson sets out the trade-offs in four dimensions, shows how to measure them for a real workload, and explains how to confirm that a batch gives answers you can trust.

The Four Dimensions

No single column wins. A chat application needs the first, a nightly classification job benefits from the second, and a large evaluation may use both. The question for each workload is which dimension matters most, and whether the other dimensions are acceptable.

Cost in Detail

Cost is the easiest dimension to calculate, and the most often estimated carelessly. The bill depends on input tokens, output tokens, and any cache reads or writes, each priced separately. For a batch, the same rates are halved, and caching discounts stack with the batch discount. The formula is the same as in Lesson 137:

def job_cost(records: int, input_tokens: int, output_tokens: int,
             input_price: float, output_price: float, batch: bool) -> float:
    factor = 0.5 if batch else 1.0
    per_record = (input_tokens * input_price + output_tokens * output_price) / 1_000_000
    return records * per_record * factor

The prices are per million tokens. The function takes them as arguments, so the same arithmetic works for any model, and the figures must come from the current pricing page.

A Worked Comparison

Suppose each classification uses 500 input tokens and 50 output tokens. The table below compares 1,000 records on two models, using the standard prices for each:

The batch saving is always half, whatever the model. The more important comparison is between models: for this workload, the batch price of Opus 5.5 equals the synchronous price of Sonnet 5.5. A team can therefore sometimes get a stronger model for batch work at the cost of a faster model for interactive work. That trade is only worth making when the stronger model's quality matters and the work can wait.

Latency and Throughput

Batch latency has two parts, and both belong in a plan. The typical case is most batches finishing within an hour, according to the documentation. The worst case is the 24-hour limit, after which unfinished requests expire, and the documentation notes that processing may slow under high demand and push more requests toward expiry.

Plan around the worst case when the work has a deadline. If a report must be ready by nine in the morning, submit the batch the previous evening and read the results on arrival, with a fallback that runs synchronously if the batch has not ended by a chosen time. Planning around the typical case alone produces missed deadlines.

Throughput is the strength of batches. A single batch holds 100,000 requests, and a job of millions of requests is simply a set of batches. Synchronous throughput is capped by concurrency and rate limits, so the same volume takes far longer that way, and more engineering is needed to keep it under the limits.

Reliability and Operations

Synchronous requests fail in the moment, and the caller sees the error immediately. Batch failures are spread across time: a request may fail only after hours of waiting, and its results appear in a file. The operational cost of that delay is real. The retry machinery from Lesson 138 is what makes batches reliable, and it is work that synchronous calls do not require in the same form.

Both approaches need monitoring, but the signals differ. A synchronous service watches latency and error rates in real time. A batch pipeline watches the status of each batch, the age of the oldest running batch, the share of requests that errored, and the count of items awaiting repair.

Choosing with a Decision Function

The trade-offs can be turned into a rule that a team agrees on in advance. The function below encodes the most common considerations: whether a person is waiting, how long the answer may take, and how large the job is:

def recommend(person_waiting: bool, hours_available: float, requests: int) -> str:
    if person_waiting:
        return "synchronous"
    if requests < 100:
        return "synchronous"          # Too small for batch overhead to pay off
    if hours_available < 24:
        return "synchronous or a split: batch for the bulk, synchronous for the deadline"
    return "batch"

The thresholds are examples, not rules. The important part is that the decision is written down, so that a new feature is judged the same way as the last one. A function like this also documents why a job runs the way it does, which matters when someone asks why a feature takes a few hours to respond.

Hybrid Designs

Many real systems use both. A common pattern is a synchronous path for the answer a person needs now, and a batch path for work that can wait:

The fallback row is the one teams forget. A batch that misses its deadline should not simply be late. A planned fallback, for the subset of requests that have not finished, turns a missed deadline into a higher cost rather than a failure.

Checking That Batches Give Trustworthy Answers

Batch and synchronous requests use the same models, so answers should agree in the same way that two synchronous calls with the same input would. They are not guaranteed to be identical, because generation varies between runs (Lesson 19). Before relying on batch results for a new task, measure agreement on a sample. Send the same records through both paths, and compare the labels or fields that matter:

def agreement(sync_answers: dict[str, str], batch_answers: dict[str, str]) -> float:
    shared = sync_answers.keys() & batch_answers.keys()
    if not shared:
        return 0.0
    matches = sum(sync_answers[key] == batch_answers[key] for key in shared)
    return matches / len(shared)

The result is a rate, not a verdict. A rate of 98% on a sample of two hundred records is strong evidence that the paths agree for practical purposes. A rate of 80% tells you the disagreements need investigation, perhaps by reading the cases where the answers differ, or by checking whether the instructions are ambiguous. The sample should be drawn at random, and it should include hard cases, since agreement on easy cases says little.

Practical Example: A Complete Decision Report

The example below combines the pieces. It costs a workload under both paths, applies the decision rule, and measures agreement on a small sample. The data is fabricated so the report is deterministic, and the same functions work on real measurements.

"""batch_vs_sync_report.py: cost, decision, and agreement for one workload."""

from batch_vs_sync import job_cost, recommend, agreement  # the functions defined in this lesson

WORKLOADS = [
    ("interactive reply", dict(person_waiting=True, hours_available=0.01, requests=1)),
    ("nightly archive", dict(person_waiting=False, hours_available=12, requests=1_000)),
    ("monthly re-scoring", dict(person_waiting=False, hours_available=72, requests=100_000)),
]

if __name__ == "__main__":
    for name, options in WORKLOADS:
        print(f"{name}: {recommend(**options)}")

    print()
    for model, input_price, output_price in [("claude-sonnet-5-5", 2.00, 10.00),
                                             ("claude-opus-5-5", 4.00, 20.00)]:
        sync = job_cost(1_000, 500, 50, input_price, output_price, batch=False)
        batch = job_cost(1_000, 500, 50, input_price, output_price, batch=True)
        print(f"{model}, 1,000 records: synchronous ${sync:.3f}, batch ${batch:.3f}")

    sync_labels = {"r1": "positive", "r2": "negative", "r3": "mixed", "r4": "mixed", "r5": "positive"}
    batch_labels = {"r1": "positive", "r2": "negative", "r3": "mixed", "r4": "positive", "r5": "positive"}
    print(f"agreement on sample: {agreement(sync_labels, batch_labels):.0%}")

To make the demonstration run as a single file, add the functions from this lesson to a module named batch_vs_sync.py, with job_cost, recommend, and agreement as written above.

Output:

interactive reply: synchronous
nightly archive: synchronous or a split: batch for the bulk, synchronous for the deadline
monthly re-scoring: batch

claude-sonnet-5-5, 1,000 records: synchronous $1.500, batch $0.750
claude-opus-5-5, 1,000 records: synchronous $3.000, batch $1.500
agreement on sample: 80%

How It Works

1. The interactive reply is synchronous, because a person is waiting. The rule returns that answer before considering size or time.
2. The nightly archive is a split, not a pure batch. The rule gives twelve hours for a thousand requests, which is less than the 24-hour limit, so the deadline is at risk, and the function recommends a split. This is the case where the fallback described above applies.
3. The monthly re-scoring is a batch. A large volume with three days of slack is the textbook case for batches.
4. The cost table uses the same function for both paths, so the 50% difference appears exactly, and the comparison between models is visible in one place.
5. The agreement figure comes from one disagreement in five records. For a sample this small, the number is a prompt to read the disagreeing case, not a conclusion about the model.

Summary

Batches trade time for money, and synchronous requests trade money for time. Cost is simple arithmetic, latency must be planned against the worst case, throughput favors batches, and operations favor simplicity. Decisions work best when they are written down, combined into hybrid designs where the workload has both urgent and patient parts, and checked against measured agreement before trust is extended. With that, the batch section is complete: the same model answers the same questions either way, and the choice is about what your workload can afford to wait for.