Handle Failed Batch Items

Failure Is Normal at Batch Scale

A batch of ten thousand requests will not succeed completely on the first attempt, and it should not be expected to. Some requests are malformed, some hit temporary service problems, some are canceled because the job itself was stopped, and some expire because the service was busy. Lesson 137 classified each outcome into one of five statuses: ok, needs_review, fix_request, retry, and not_run. This lesson decides what to do with the three statuses that lead to another attempt, and designs the workflow that carries work through repeated rounds until it is finished or deliberately abandoned.

The guiding idea is that a failed item is not a crisis. It is a record with a status, a count of attempts, and a next action. Treating failures as data makes them manageable.

Two Causes, Two Responses

The most important distinction is between failures caused by the request and failures caused by the environment:

Resubmitting an invalid request unchanged wastes money only if it runs, and the batch's invalid-request errors appear at the end of processing, so the time cost is the real problem. Fixing the request first avoids repeating the same failure for hours.

Keeping Attempt Counts

Retries must be bounded. A request that fails for a reason no retry can solve should not circulate forever. The registry from Lesson 133 is the right place for an attempt counter, because it survives crashes and is visible to every process:

ATTEMPTS_DDL = """
ALTER TABLE batch_items ADD COLUMN attempts INTEGER NOT NULL DEFAULT 0
"""

Each time a request is placed in a batch, its attempts value increases by one. A request that reaches the limit moves to a dead-letter state: it is kept with its last error, and it waits for a person, rather than being submitted again automatically.

The limit should match the kind of failure. Temporary problems justify a few attempts, because the next round may find the service healthy. Invalid requests justify fewer automatic attempts, because a retry without a fix cannot succeed.

Building the Retry Plan

A retry round is a batch built from the registry: every request whose latest status calls for another attempt, and which has not reached its attempt limit. Building it from the registry, rather than from the previous batch's file, means the plan always reflects the current state, even after several rounds:

from dataclasses import dataclass, field


@dataclass
class RetryPlan:
    to_resubmit: list[str] = field(default_factory=list)    # custom_ids for the next batch
    needs_fix: list[str] = field(default_factory=list)      # invalid requests, for a person or a repair step
    exhausted: list[str] = field(default_factory=list)      # reached the attempt limit


def plan_retries(rows: list[dict], max_attempts: int = 3) -> RetryPlan:
    plan = RetryPlan()
    for row in rows:
        if row["status"] == "fix_request":
            plan.needs_fix.append(row["custom_id"])
        elif row["status"] in ("retry", "not_run"):
            if row["attempts"] >= max_attempts:
                plan.exhausted.append(row["custom_id"])
            else:
                plan.to_resubmit.append(row["custom_id"])
    return plan

The plan separates the three destinations clearly. Items in to_resubmit go into the next batch unchanged, items in needs_fix are held back until their requests are repaired, and items in exhausted are reported. Nothing is silently dropped.

Repairing Invalid Requests

Some invalid requests have a mechanical fix. A common example is a max_tokens value above what the model accepts, which can be reduced to the limit. Repairs should be explicit, narrow, and logged, because a repair changes what the model is asked to do. The repair function below handles only one documented case, and returns None for everything else, so unknown problems still reach a person:

MODEL_MAX_OUTPUT = {"claude-sonnet-5-5": 128_000}


def repair(request: dict, error_message: str) -> dict | None:
    """Return a corrected request, or None if this repair does not apply."""
    params = request["params"]
    if "max_tokens" in error_message and "too large" in error_message:
        limit = MODEL_MAX_OUTPUT.get(params["model"])
        if limit and params["max_tokens"] > limit:
            fixed = {**request, "params": {**params, "max_tokens": limit}}
            return fixed
    return None

Notice that the repair builds a new dictionary instead of changing the original. The original request remains available in the registry, so you can always see what was sent and what was changed. A repaired request should also be recorded with its change, because a reviewer may want to know that a limit was lowered.

Testing a Repaired Request Synchronously

Batch validation is slow, so a repaired request should be checked before it joins a large batch. The synchronous Messages API returns a result immediately, which means one test call confirms the shape:

def check_with_sync_api(client, request: dict) -> bool:
    try:
        client.messages.create(**request["params"])
        return True
    except Exception:
        return False

A single successful test call does not prove the whole batch will succeed, but it removes the most common cause of repeated failure. The batch can then be submitted with confidence.

Batch-Level Failures

Sometimes the whole batch fails rather than individual items. The most common cause is size: a batch over the documented 256 MB limit is rejected, and the documentation notes that this can appear as a 413 request_too_large error. The response is not a retry but a split, using the builder from Lesson 132, which keeps each batch under the safe limits. Batch-level cancellation is the other case, handled by the status checks from Lesson 134: a canceled batch's unprocessed items return as canceled, and they flow into the same retry plan as any other temporary failure.

Practical Example: Three Rounds of Retries

The example below simulates a job across three rounds. The first round has a temporary failure, an invalid request that can be repaired, and an invalid request that cannot. The second and third rounds retry the temporary failure. The final check runs the planner once more, after the attempts have been counted, to show the attempt limit taking effect. The demonstration uses in-memory rows, so it runs offline.

"""retry_rounds.py: plan retries from registry rows, repair what can be repaired, and stop at a limit."""

from dataclasses import dataclass, field

MODEL_MAX_OUTPUT = {"claude-sonnet-5-5": 128_000}


@dataclass
class RetryPlan:
    to_resubmit: list[str] = field(default_factory=list)
    needs_fix: list[str] = field(default_factory=list)
    exhausted: list[str] = field(default_factory=list)


def plan_retries(rows: list[dict], max_attempts: int = 3) -> RetryPlan:
    plan = RetryPlan()
    for row in rows:
        if row["status"] == "fix_request":
            plan.needs_fix.append(row["custom_id"])
        elif row["status"] in ("retry", "not_run"):
            if row["attempts"] >= max_attempts:
                plan.exhausted.append(row["custom_id"])
            else:
                plan.to_resubmit.append(row["custom_id"])
    return plan


def repair(request: dict, error_message: str) -> dict | None:
    params = request["params"]
    if "max_tokens" in error_message and "too large" in error_message:
        limit = MODEL_MAX_OUTPUT.get(params["model"])
        if limit and params["max_tokens"] > limit:
            return {**request, "params": {**params, "max_tokens": limit}}
    return None


def new_round(rows: list[dict], requests: dict[str, dict], errors: dict[str, str]):
    plan = plan_retries(rows)
    repaired = []
    for custom_id in plan.needs_fix:
        fixed = repair(requests[custom_id], errors[custom_id])
        if fixed:
            requests[custom_id] = fixed
            repaired.append(custom_id)
    for row in rows:
        if row["custom_id"] in plan.to_resubmit + repaired:
            row["attempts"] += 1
    return plan, repaired


if __name__ == "__main__":
    requests = {
        "review-1": {"custom_id": "review-1", "params": {"model": "claude-sonnet-5-5", "max_tokens": 256}},
        "review-2": {"custom_id": "review-2", "params": {"model": "claude-sonnet-5-5", "max_tokens": 200_000}},
        "review-3": {"custom_id": "review-3", "params": {"model": "claude-sonnet-5-5", "max_tokens": 256}},
    }
    errors = {"review-2": "max_tokens is too large", "review-3": "unsupported option: top_k"}
    rows = [
        {"custom_id": "review-1", "status": "retry", "attempts": 0},          # temporary failure
        {"custom_id": "review-2", "status": "fix_request", "attempts": 0},    # repairable
        {"custom_id": "review-3", "status": "fix_request", "attempts": 0},    # not repairable
    ]

    plan, repaired = new_round(rows, requests, errors)
    print("round 1 resubmit:", plan.to_resubmit, "| repaired:", repaired, "| needs fix:", plan.needs_fix)
    print("round 1 repaired max_tokens:", requests["review-2"]["params"]["max_tokens"])

    for _ in range(2):                                # Rounds 2 and 3: review-1 fails again
        new_round(rows, requests, errors)

    final = plan_retries(rows)
    print("after three rounds, attempts:", rows[0]["attempts"], "| exhausted:", final.exhausted)

Output:

round 1 resubmit: ['review-1'] | repaired: ['review-2'] | needs fix: ['review-2', 'review-3']
round 1 repaired max_tokens: 128000
after three rounds, attempts: 3 | exhausted: ['review-1']

How It Works

1. Round one builds the plan from the rows. review-1 is a temporary failure, so it is queued for resubmission. Both fix_request rows land in needs_fix, because the plan is made before any repair happens.
2. The repair applies only to review-2. Its max_tokens is above the model's limit, so it is lowered to 128,000. The request is replaced in the request map, and the error record keeps the original message.
3. review-3 cannot be repaired. Its error names an unsupported option, which the repair function does not recognize, so it stays in needs_fix for a person.
4. Attempts increase only for items that are actually resubmitted or repaired. review-3 is not counted, because it has not been sent.
5. Rounds two and three each count one attempt for review-1, reaching three in total. The final planning pass runs after those increments, so review-1 has reached the limit and appears in exhausted. The order matters: a plan made before the last increment would still show it as eligible.

Note: the demonstration uses a single attempt counter for both temporary failures and repairs. In a production system, repairs and retries may deserve separate limits, because a repair changes the request while a retry does not.

The Full Lifecycle

Putting the pieces together gives one loop per round: read the registry, plan the retries, repair what is repairable, check repaired requests synchronously, build the next batch, submit it, wait for it (Lesson 134), download its results (Lesson 136), process them (Lesson 137), and write the new statuses back to the registry. Each round starts from the registry rather than from memory, so an interruption at any point loses nothing: the next run picks up exactly where the registry says the work stands.

The loop ends when every item is ok, or when every remaining item is needs_fix or exhausted. At that point the job is finished in the only sense that matters: every item has either a result or a recorded reason for not having one.