Handle Rate Limits

When Too Much Is Happening at Once

A rate limit is the service's way of saying that a caller has sent too much work in a given period. It is the most common temporary error an application meets, and in many systems the first one to appear as traffic grows. The response is a 429 status, and the SDK raises a RateLimitError (Lesson 141).

Unlike a 400, a rate limit is temporary by design. The limit resets as time passes, so the right response is to slow down, not to change the request. This lesson explains what the limits measure, how the service tells the caller when to try again, and how to design an application that stays under its limits instead of constantly bumping into them.

What Limits Measure

Rate limits are not a single number. The documentation describes several dimensions, and a request can hit any of them:

The values depend on the model and the account's usage tier, and they change over time. The rate limits page in the documentation has the current figures for each tier. Designs should be based on the limits that apply to the account in use, and not on figures from memory.

Limits apply to the account and workspace, not to a single process. Ten workers sharing one key draw from the same allowance, which is why rate limiting is a concern for the whole application, not each function alone.

Some limits behave differently from throughput limits. The documentation describes spend limits, which can be reached when an account or workspace has spent its allowance. A spend limit does not clear as time passes, and it is signaled differently from a throughput limit, so waiting and retrying cannot solve it. The error's details say which kind of limit was reached, and a handler should check that before deciding to retry.

The Retry-After Header

When the service rejects a request for throughput reasons, it often includes a retry-after header that says how many seconds to wait before trying again. The value is the most accurate guidance available, because the service knows when its window resets:

HTTP/1.1 429 Too Many Requests
retry-after: 12
request-id: req_011...

A caller that waits for exactly that long, and no longer, recovers as quickly as possible. A caller that ignores the header and retries immediately will be rejected again, and may make the situation worse for every other caller sharing the key.

The header can take more than one form, and the most common is a whole number of seconds. The function below reads that form, and ignores anything it does not recognize, so the caller can fall back to its own backoff:

def retry_after_seconds(headers) -> float | None:
    value = headers.get("retry-after")
    if value is None:
        return None
    try:
        seconds = float(value)
    except ValueError:
        return None           # An HTTP date, for example, is not handled here
    return seconds if seconds > 0 else None

Returning None for unrecognized values is deliberate. A missing or unusable header should not stop the program. It should lead to a fallback, which Lesson 146 covers.

Rate-Limit Headers

Successful responses also carry headers that describe the account's current limits and remaining allowance. Reading them lets an application slow down before it is rejected. The documentation lists the exact header names for each limit type on the rate limits page, and they are worth checking there before use, since names can change. The principle is the same regardless of the names: track what remains, and pace work so the remaining allowance is not exhausted.

Because the SDK's response objects expose headers through the raw response, a program that needs them can read them from the response wrapper, instead of only from the parsed message.

Pacing with a Token Bucket

Waiting for a 429 is reactive. Pacing is proactive: the application decides how fast to send, so the limit is rarely reached. The standard tool for this is the token bucket. Imagine a bucket that fills with permission at a steady rate, up to a capacity. Each request takes one or more tokens from the bucket. When the bucket is empty, the caller waits for enough permission to accumulate.

The implementation below works with any clock, so the demonstration can control time and produce exact waits:

"""token_bucket.py: pace requests so they stay within an allowance."""


class TokenBucket:
    def __init__(self, rate_per_second: float, capacity: float, clock):
        self.rate = rate_per_second
        self.capacity = capacity
        self.tokens = capacity
        self.clock = clock
        self.last = clock()

    def take(self, amount: float = 1.0) -> float:
        """Consume `amount` and return how many seconds the caller must wait first."""
        now = self.clock()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        self.tokens -= amount
        if self.tokens >= 0:
            return 0.0
        return -self.tokens / self.rate


if __name__ == "__main__":
    now = [0.0]
    bucket = TokenBucket(rate_per_second=2.0, capacity=2.0, clock=lambda: now[0])
    waits = [bucket.take() for _ in range(4)]     # Four requests at the same instant
    print("waits:", waits)

Output:

waits: [0.0, 0.0, 0.5, 1.0]

How It Works

1. The bucket starts full, with capacity for two requests at once. The first two requests need no waiting.
2. Each take() removes one token, which may make the balance negative. A negative balance is a debt, and the wait is the time needed to pay it back at the refill rate.
3. The third request leaves the bucket at −1 tokens. At two tokens per second, the caller must wait half a second.
4. The fourth request leaves the balance at −2, so it waits a full second. The waits add up because each request is queued behind the ones before it.
5. Time moves the balance back up. If the caller waits and calls again, the refill is added before the debt is computed, so the bucket recovers on its own.

The bucket is shared. Every worker in the process uses the same instance, and the instance is protected by a lock when workers are threads. That shared accounting is what makes the limit a property of the application, not of one call site.

Combining Pacing and Waiting

Pacing keeps most requests under the limit. Reactive waiting handles the rest, because limits can be reached in ways the bucket does not model, such as other applications using the same key. A complete handler does both: it takes permission from the bucket before each request, and if a 429 still arrives, it waits for the server's retry-after value, or falls back to backoff.

def paced_call(bucket, send, sleep):
    sleep(bucket.take())                    # Proactive: wait for our own allowance
    try:
        return send()
    except RateLimitError as error:         # Reactive: the server says stop
        delay = retry_after_seconds(error.response.headers) or 1.0
        sleep(delay)
        raise                                # The caller decides whether to try again

The function re-raises after waiting, rather than retrying silently. Whether a retry is appropriate depends on the caller's policy, which Lesson 143 covers in detail.

Concurrency as a Separate Limit

Rate limits count requests and tokens over time. A separate concern is how many requests are in flight at once. Sending a hundred requests simultaneously can exhaust the requests-per-minute allowance in a second, even when the average rate is acceptable. A concurrency limit caps the number of simultaneous calls, which smooths bursts. Lesson 75 showed the same idea with asynchronous tasks, and the same semaphore pattern applies to threads.

Practical Example: A Rate-Limited Worker Pool

The example below runs ten requests through a fake service that accepts at most three requests in any one-second window, first without pacing and then with it. The paced run uses a rate slightly below the limit, which leaves headroom for other callers and avoids borderline cases at the edge of the window. The clock is simulated, so the output is deterministic.

"""rate_limited_pool.py: a fake rate-limited service, with and without client-side pacing."""

from token_bucket import TokenBucket


class FakeService:
    """Accepts at most `limit` requests in any one-second window, and rejects the rest."""

    def __init__(self, limit: int):
        self.limit = limit
        self.accepted_at: list[float] = []
        self.rejected = 0

    def send(self, now: float) -> bool:
        recent = [t for t in self.accepted_at if now - t < 1.0]
        if len(recent) < self.limit:
            self.accepted_at.append(now)
            return True
        self.rejected += 1
        return False


def run(paced: bool, requests: int = 10, limit: int = 3) -> tuple[int, float]:
    now = [0.0]
    service = FakeService(limit)
    bucket = TokenBucket(rate_per_second=limit * 0.9, capacity=1.0, clock=lambda: now[0])
    for _ in range(requests):
        if paced:
            now[0] += bucket.take()             # Wait for permission before sending
        service.send(now[0])
    return service.rejected, now[0]


if __name__ == "__main__":
    rejected, finished = run(paced=False)
    print(f"without pacing: {rejected} rejected, finished at {finished:.1f}s")
    rejected, finished = run(paced=True)
    print(f"with pacing:    {rejected} rejected, finished at {finished:.1f}s")

Output:

without pacing: 7 rejected, finished at 0.0s
with pacing:    0 rejected, finished at 3.3s

How It Works

1. The fake service uses a sliding window. A request is accepted if fewer than three accepted requests fall within the last second. This is closer to how rate limits behave than a window that resets at fixed times.
2. Without pacing, all ten requests arrive at the same instant. Three succeed, and seven are rejected, which is what happens when a burst meets a limit.
3. The bucket has capacity one, so it allows no initial burst. Its refill rate is 2.7 requests per second, which is 90% of the limit, so requests are spaced about 0.37 seconds apart.
4. At that spacing, a one-second window never holds more than two earlier requests, so every new request is accepted. The margin is deliberate: at exactly three per second, a request could fall on the edge of the window, where rounding decides the outcome.
5. The cost of pacing is time. The tenth request is sent at 9 ÷ 2.7, about 3.3 seconds. The unpaced run finishes instantly, but only three of its requests did any work.
6. Against a real service, the same margin matters for the same reason: other callers share the allowance, so planning at exactly the published limit leaves no room for them.

Summary

Rate limits are temporary, measured across several dimensions, and shared by everything using the same account. The service's retry-after value is the most accurate signal for when to resume. Pacing with a token bucket keeps an application under its allowance, and reactive handling covers what pacing cannot predict. Spend limits are a different kind of limit, which waiting does not clear, and the error details are what distinguish them. The next lesson turns the retry decision into a complete policy.