Build Reliable API Calls

Bringing the Section Together

Lessons 141–149 covered the parts of reliability one at a time: classifying errors, handling rate limits, deciding on retries, setting timeouts, recovering from network failures, computing backoff, logging, tracing request IDs, and paginating lists. Each part is necessary, and none is sufficient alone. A call that retries without a timeout can hang for half an hour. A call that paces perfectly but never logs cannot be investigated. A call that retries forever keeps hammering a service that is down.

This lesson assembles the parts into one function, reliable_call(), that a larger application can use for every request to the model. It adds one component the earlier lessons did not yet need: a circuit breaker, which stops sending requests entirely when a service has clearly failed, and resumes later. The result is a single entry point whose behavior can be read in one place, and tested in one place.

The Design

A reliable call has a fixed sequence of decisions, and the order matters. Before anything is sent, the call asks whether the service is worth trying at all. If it is, it paces the request, sends it, and then classifies the outcome. Each outcome leads to one of four results: success, a terminal error that waiting cannot fix, a retry after a delay, or a give-up after the last attempt.

circuit open? ──yes──► stop, do not send
     │no
     ▼
pace (token bucket) ──► send ──► success ──► record success, return answer
                          │
                          └─► error ──► terminal? ──yes──► return error, leave breaker alone
                                             │no
                                             ▼
                                  attempts left? ──yes──► wait (backoff), go back to send
                                             │no
                                             ▼
                                  record failure with breaker, return error

The decisions in this flow come from earlier lessons, and the new piece is the breaker. Everything else is composition.

The Circuit Breaker

A retry handles a single request that failed. A circuit breaker handles a pattern: many requests failing in a row, which suggests the service itself is unhealthy. Continuing to send requests in that state wastes time and adds load. The breaker watches consecutive failures, and when they reach a threshold, it opens, and refuses to send anything for a cooldown period. After the cooldown, it allows a request through to test whether the service has recovered. A success closes it again, and a failure reopens it.

The name comes from electrical circuit breakers, which cut the current when it becomes dangerous, and restore it after the fault has been cleared. The implementation below keeps the three states implicit, through the failure count and the time the breaker opened:

"""circuit_breaker.py: stop sending requests while a service is clearly failing."""


class CircuitBreaker:
    def __init__(self, threshold: int, cooldown: float, clock):
        self.threshold = threshold
        self.cooldown = cooldown
        self.clock = clock
        self.failures = 0
        self.opened_at: float | None = None

    def allow(self) -> bool:
        if self.opened_at is None:
            return True                                  # Closed: send normally
        return self.clock() - self.opened_at >= self.cooldown   # Cooldown over: allow a trial

    def record_success(self) -> None:
        self.failures = 0
        self.opened_at = None

    def record_failure(self) -> None:
        self.failures += 1
        if self.failures >= self.threshold:
            self.opened_at = self.clock()

    @property
    def is_open(self) -> bool:
        return self.opened_at is not None and not self.allow()

How It Works

1. Closed is the normal state. With no recorded open time, allow() returns true, and failures are counted only as they happen.
2. The breaker opens when failures reach the threshold. record_failure() stores the time of the failure that tipped it over. From then on, allow() returns false until the cooldown has elapsed.
3. After the cooldown, allow() returns true again, which lets one request through as a trial. The breaker does not need a separate state for this: the time comparison does the work.
4. A success clears everything, closing the breaker and resetting the count. A failure during a trial sets a new open time, so the breaker waits another full cooldown.
5. Only service-side failures should count. A 400 says the request is wrong, not that the service is down, so the caller decides which failures reach record_failure(). The function below follows that rule.

Assembling the Call

The reliable call combines the breaker, the retry policy, the backoff calculation, and the trace from Lesson 148. It takes the send function as an argument, so the same logic works with the live client or with a test double:

"""reliable_call.py: one entry point combining retries, backoff, breaker, and tracing."""

from dataclasses import dataclass
from typing import Any

from anthropic import APIConnectionError, APIError, APIStatusError

from backoff import backoff_delay
from circuit_breaker import CircuitBreaker
from request_trace import Trace

RETRYABLE_STATUS = {408, 409, 429}


def is_retryable(error: Exception) -> bool:
    if isinstance(error, APIConnectionError):
        return True
    if isinstance(error, APIStatusError):
        return error.status_code in RETRYABLE_STATUS or error.status_code >= 500
    return False


@dataclass
class CallResult:
    ok: bool
    value: Any
    note: str
    sends: int
    trace: Trace


def reliable_call(send, *, trace: Trace, breaker: CircuitBreaker, sleep,
                  max_attempts: int = 4, base: float = 0.5, cap: float = 8.0) -> CallResult:
    sends = 0
    for attempt in range(1, max_attempts + 1):
        if not breaker.allow():
            trace.record(None, "circuit open, not sent")
            return CallResult(False, None, "circuit open, not sent", sends, trace)

        sends += 1
        try:
            value = send()
        except APIError as error:
            name = type(error).__name__
            request_id = getattr(error, "request_id", None)
            if not is_retryable(error):
                trace.record(request_id, f"stopped: {name}")
                return CallResult(False, None, f"stopped: {name}", sends, trace)
            if attempt == max_attempts:
                trace.record(request_id, f"gave up: {name}")
                breaker.record_failure()
                return CallResult(False, None, f"gave up: {name}", sends, trace)
            trace.record(request_id, f"retrying after {name}")
            sleep(backoff_delay(attempt - 1, base=base, cap=cap))
            continue

        trace.record(None, "answered")
        breaker.record_success()
        return CallResult(True, value, "answered", sends, trace)

    raise AssertionError("unreachable: the loop always returns")

How It Works

1. The breaker is consulted before every send, including retries. A breaker that opens during a retry sequence stops that sequence too, which is the point of having one.
2. Each send is counted, so the caller can tell how many requests actually left the process. A call stopped by the breaker has zero sends.
3. A terminal error returns immediately and does not touch the breaker. The request was wrong, so the service is not implicated.
4. A retryable error waits and loops, with the delay from Lesson 146. The wait is computed from the attempt number, so the schedule matches the earlier examples.
5. The final failure feeds the breaker. Only exhausted retries count, because a single transient failure that a retry resolved is not a sign of a broken service.
6. Every path records the trace, so the outcome of every call can be explained afterward, including the calls the breaker refused.

Note: send here is any function that makes one request and raises on failure. In production it is a lambda around client.messages.create(...), and it is best to set the SDK's own max_retries to zero, so that retries happen in one place, as Lesson 143 recommended.

Practical Example: Five Calls Against a Failing Service

The example below runs five calls through the reliable wrapper, against a scripted service that succeeds once, fails permanently once, and fails repeatedly twice. The clock and the sleep function are simulated, so the waits and the cooldown are exact. The output shows what each call did, and when the breaker stopped the fifth call.

"""reliable_demo.py: five calls through the reliable wrapper, against a scripted service."""

from api_errors import fake_status_error
from circuit_breaker import CircuitBreaker
from reliable_call import reliable_call
from request_trace import Trace


def script(outcomes):
    queue = list(outcomes)

    def send():
        outcome = queue.pop(0)
        if isinstance(outcome, Exception):
            raise outcome
        return outcome
    return send


def err(status, error_type, message, request_id):
    return fake_status_error(status, error_type, message, request_id)


if __name__ == "__main__":
    now = [0.0]
    clock = lambda: now[0]
    sleep = lambda seconds: now.__setitem__(0, now[0] + seconds)
    breaker = CircuitBreaker(threshold=2, cooldown=30.0, clock=clock)

    calls = [
        [err(529, "overloaded_error", "overloaded", "req_a1"),
         err(429, "rate_limit_error", "rate limit", "req_a2"), "answer"],
        [err(400, "invalid_request_error", "bad input", "req_b1")],
        [err(503, "api_error", "unavailable", f"req_c{n}") for n in range(1, 5)],
        [err(503, "api_error", "unavailable", f"req_d{n}") for n in range(1, 5)],
        [],
    ]

    results = []
    for number, outcomes in enumerate(calls, start=1):
        trace = Trace(trace_id=f"call-{number}", operation="summarize")
        result = reliable_call(script(outcomes), trace=trace, breaker=breaker, sleep=sleep,
                               max_attempts=4, base=0.5, cap=8.0)
        results.append(result)
        print(f"call {number}: ok={result.ok} sends={result.sends} note={result.note}")
    print("breaker open after call 4:", breaker.is_open)
    print(results[2].trace.support_summary())

Output:

call 1: ok=True sends=3 note=answered
call 2: ok=False sends=1 note=stopped: BadRequestError
call 3: ok=False sends=4 note=gave up: InternalServerError
call 4: ok=False sends=4 note=gave up: InternalServerError
call 5: ok=False sends=0 note=circuit open, not sent
breaker open after call 4: True
trace call-3 (summarize)
  attempt 1: request_id=req_c1 retrying after InternalServerError
  attempt 2: request_id=req_c2 retrying after InternalServerError
  attempt 3: request_id=req_c3 retrying after InternalServerError
  attempt 4: request_id=req_c4 gave up: InternalServerError

How It Works

1. Call one recovers. The overload and the rate limit are both retried, with delays of half a second and one second, and the third send succeeds. Three sends, one answer.
2. Call two stops at once. The bad request is terminal, so there is one send, no waiting, and no effect on the breaker.
3. Call three exhausts its retries. Four sends, with waits of half a second, one second, and two seconds in between. The breaker counts one failure, which is below the threshold of two, so it stays closed.
4. Call four exhausts its retries too, which is the second consecutive failure. The breaker opens at that moment.
5. Call five is refused without a send. The breaker is open, and the cooldown of thirty seconds has not elapsed, so the call returns immediately with zero sends.
6. The trace for call three shows every attempt. A support engineer reading it sees the four request IDs, the outcome of each, and that the final attempt was given up on, which is the information needed to investigate.

Putting It Into an Application

A production application would wrap reliable_call() in a small service object, with one breaker per upstream service, and one token bucket shared across the process (Lesson 142). The function itself stays the same. The service object provides the send function, creates a trace per user action, and writes the trace to the structured log (Lessons 147 and 148). The breaker's state is then visible to monitoring, so an operator can see that a service is in a cooldown before users start reporting failures.

Summary

A reliable call is a composition: pacing keeps the request inside the allowance, the timeout bounds the wait, the retry layer handles transient failures with backoff, the breaker stops traffic to a failing service, the logs and traces make every outcome explainable, and the pagination and error classes from earlier lessons cover the remaining cases. Each piece is small and testable, and the wrapper makes their order explicit. Reliability does not come from any one of these techniques. It comes from applying them together, in a fixed order, to every call.