Retry Failed Requests

A Retry Is a Second Chance, Not a Guarantee

Lesson 141 showed that errors fall into groups: some call for a fix, and some call for waiting. Retrying is the response to the second group. When a request fails for a reason that may pass, sending it again often succeeds, and the caller's work is saved without anyone noticing.

Retrying also has costs and risks. Each attempt takes time and may be billed. A retry can multiply load on a service that is already struggling. And for some operations, repeating a request is not the same as doing it once. This lesson covers the decision of whether to retry, the difference between the SDK's built-in retries and a retry layer of your own, and the idempotency questions that decide whether a retry is safe.

Which Errors Are Worth Retrying

The SDK's own policy, described in the documentation, retries these cases by default: connection errors, timeouts, 408 (request timeout), 409 (conflict), 429 (rate limit), and any status of 500 and above. Anything else, such as 400, 401, 403, 404, and 422, is not retried, because waiting does not change those outcomes.

The SDK also honors a special response header, x-should-retry. When the server sets it to true or false, that value overrides the default rule. A server that knows a request will never succeed can say so, and a server that expects recovery can invite a retry. A handler that writes its own policy should check this header first, since it carries the server's direct instruction.

The policy above is a starting point, and the decision still depends on the operation. Retrying a read is harmless. Retrying a write may not be, as the idempotency section below explains.

The SDK's Built-In Retries

The client retries automatically, with no code of your own. The documentation and the SDK source give these defaults:

The delay grows with each attempt, and includes randomness so that many clients do not retry in lockstep. Lesson 146 explains the pattern in detail. When the server sends a retry-after value, the SDK uses that value rather than its own calculation, within limits.

Configuration can be set once for the whole client, or per request:

from anthropic import Anthropic

client = Anthropic(max_retries=5)                  # More patience for every call
client = Anthropic(max_retries=0)                  # Disable the built-in retries entirely
response = client.with_options(max_retries=4).messages.create(...)   # One call only

The built-in retries cover many transient problems with no code. Their limit is that they are tuned for a single request. A long-running job may need a policy that considers the whole job, and that is the reason for building a retry layer of your own.

Stacking Retries

The SDK retries inside each call, and an application may retry around each call as well. The two multiply. If the SDK makes three attempts and the application makes four calls, each of which the SDK retries, the total number of HTTP requests can reach twelve for a single unit of work. That is rarely what anyone intended, and it can turn a brief outage into a flood of traffic.

The usual solution is to choose one layer. Either the SDK retries, with its default or a configured count, and the application makes one call, or the SDK retries are disabled and the application owns the whole policy. The second option is what the rest of this section uses, because it allows one place to count attempts, log them, and decide when to stop.

The arithmetic is simple enough to check in code:

def worst_case_requests(sdk_retries: int, app_attempts: int) -> int:
    return (sdk_retries + 1) * app_attempts


print(worst_case_requests(sdk_retries=2, app_attempts=4))   # Both layers active
print(worst_case_requests(sdk_retries=0, app_attempts=4))   # One layer only

Output:

12
4

Idempotency: The Question Behind Every Retry

An operation is idempotent if doing it twice has the same effect as doing it once. Reading a batch's status is idempotent, since it changes nothing. A message request is more subtle, and the subtlety matters.

Consider a request that the server processes successfully, but whose response is lost on the way back. The caller sees a timeout and retries. The server processes the second request too. The caller now has two answers, and has been billed for both. For a question-and-answer call, this is wasteful but harmless. For an operation with side effects, it can be a serious problem.

This is why the answer to "is it safe to retry?" depends on what the request does:

A guard can be as simple as recording the intent before acting, and checking the record before retrying. A tool that books a session should record the booking key first, so a repeated attempt finds the existing booking instead of creating another.

Designing an Application-Level Retry Layer

With the SDK's retries disabled, the application's policy has four parts: which errors to retry, how many attempts to make, how long to wait between them, and what to record. The decision function below covers the first part. It follows the SDK's policy and adds one rule the SDK cannot know: the caller's own limit on attempts.

"""retry_policy.py: an application-level retry layer around SDK calls."""

import time

from anthropic import APIConnectionError, APIError, APIStatusError

RETRYABLE_STATUS = {408, 409, 429}


def should_retry(error: Exception) -> bool:
    if isinstance(error, APIConnectionError):           # Includes timeouts
        return True
    if isinstance(error, APIStatusError):
        return error.status_code in RETRYABLE_STATUS or error.status_code >= 500
    return False


def call_with_retries(fn, log: list[str], max_attempts: int = 4,
                      base: float = 0.5, cap: float = 8.0, sleep=time.sleep):
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except APIError as error:
            name = type(error).__name__
            if not should_retry(error) or attempt == max_attempts:
                log.append(f"attempt {attempt}: {name} -> stop")
                raise
            delay = min(cap, base * 2 ** (attempt - 1))
            log.append(f"attempt {attempt}: {name} -> wait {delay:g}s")
            sleep(delay)

How It Works

1. should_retry() implements the SDK's rule: connection problems and timeouts, the three retryable status codes, and every status of 500 and above. A 400 or 401 returns False immediately.
2. call_with_retries() counts attempts, and stops when the policy says no or when the attempt limit is reached. The last attempt re-raises, so the caller sees the real error instead of a silent failure.
3. The wait doubles each time, capped at eight seconds, which matches the SDK's own range. Lesson 146 explains why the doubling is useful.
4. The log records every decision, so a person can see afterward why a call took as long as it did, or why it gave up.
5. sleep is a parameter, so tests can record the waits instead of performing them.

Practical Example: Three Retry Scenarios

The example below runs the retry layer against three scripted services: one that recovers after two transient failures, one that fails with a permanent error, and one that never recovers. Each run prints its decision log, so the behavior is visible without any network calls.

"""retry_scenarios.py: the retry layer against scripted services."""

from api_errors import fake_status_error
from retry_policy import call_with_retries


def scripted(outcomes):
    """Return a function that yields each scripted outcome in turn."""
    queue = list(outcomes)

    def call():
        outcome = queue.pop(0)
        if isinstance(outcome, Exception):
            raise outcome
        return outcome
    return call


def run(label, outcomes):
    log: list[str] = []
    waits: list[float] = []
    try:
        result = call_with_retries(scripted(outcomes), log, sleep=waits.append)
        log.append(f"result: {result}")
    except Exception as error:
        log.append(f"raised: {type(error).__name__}")
    print(f"{label}:")
    for line in log:
        print("  " + line)


if __name__ == "__main__":
    run("transient, then recovers", [
        fake_status_error(529, "overloaded_error", "overloaded", "req_a"),
        fake_status_error(429, "rate_limit_error", "rate limit", "req_b"),
        "ok",
    ])
    run("permanent error", [
        fake_status_error(400, "invalid_request_error", "bad input", "req_c"),
    ])
    run("never recovers", [
        fake_status_error(503, "api_error", "unavailable", "req_d"),
        fake_status_error(503, "api_error", "unavailable", "req_e"),
        fake_status_error(503, "api_error", "unavailable", "req_f"),
        fake_status_error(503, "api_error", "unavailable", "req_g"),
    ])

Output:

transient, then recovers:
  attempt 1: InternalServerError -> wait 0.5s
  attempt 2: RateLimitError -> wait 1s
  result: ok
permanent error:
  attempt 1: BadRequestError -> stop
  raised: BadRequestError
never recovers:
  attempt 1: InternalServerError -> wait 0.5s
  attempt 2: InternalServerError -> wait 1s
  attempt 3: InternalServerError -> wait 2s
  attempt 4: InternalServerError -> stop
  raised: InternalServerError

How It Works

1. The first scenario recovers. The overloaded 529 is treated as a server error, so it is retried after half a second. The rate limit is retried after a second, and the third attempt succeeds. Nothing in the calling code needed to know about either failure.
2. The second scenario stops immediately. A 400 means the request itself is wrong, and waiting would not help, so the layer raises on the first attempt.
3. The third scenario exhausts its budget. Four attempts are allowed, and the waits double each time: half a second, one second, and two seconds. The fourth failure is raised rather than swallowed, so the caller can record it as a failure.
4. The logs explain every outcome, which is the difference between a retry layer that people trust and one they must debug blindly.
5. Waits are recorded, not performed. The waits list is the value passed as sleep, so the same tests run in milliseconds while still checking the timing.

Retrying Within a Batch of Work

A retry layer around one call protects that call. A job that makes thousands of calls needs the same policy applied to each one, plus a limit on the total effort. Without that limit, a service outage can make every call retry in full, and the job spends hours on attempts that were always going to fail. Lesson 96 introduced budgets for tool calls, and the same idea applies here: a total number of retries for the job, reported when it is used up, so the job stops and reports rather than continuing to spend.

Summary

Retry the errors that waiting can fix, and only those. The SDK retries by default, with a sensible policy, and an application that needs more control turns those retries off and owns the policy, so that attempts are counted once. Idempotency decides whether a retry is safe, because a repeated request can repeat its side effects. The decision function and the retry layer make the policy explicit, and the logs make it accountable. The next lesson looks at the other half of the timing question: how long to wait before giving up.