Two Different Kinds of Limit

Lesson 1 grouped 429 responses under "retryable errors" without distinguishing why a request might actually be rate-limited. There are two meaningfully different reasons, and they call for different responses: a rate limit (too many requests, or too much volume, in a given time window) is a transient condition that clears as time passes, while a spend limit (a hard cap on total spending reached) is a condition that a retry — however well-backed-off — will never resolve, since no amount of waiting increases a budget that's already been fully used. Confusing the two leads to exactly the wrong response in each direction: retrying a spend-limit error forever, or treating a rate limit as if it required a permanent, structural fix.

What a Rate Limit Actually Limits

A rate limit is typically expressed across more than one dimension at once — requests per minute, and tokens per minute, are both common, and either one can be the actual constraint being hit, independent of the other.

def estimate_which_limit_is_binding(requests_per_minute: int, tokens_per_minute: int, request_limit: int, token_limit: int) -> str:
    """Illustrative — confirm your account's actual current rate limits
    against your platform account's dashboard, since limits vary by usage
    tier and can be adjusted over time."""
    request_utilization = requests_per_minute / request_limit
    token_utilization = tokens_per_minute / token_limit
    if token_utilization > request_utilization:
        return "token-per-minute limit is the binding constraint"
    return "requests-per-minute limit is the binding constraint"

print(estimate_which_limit_is_binding(50, 900_000, 500, 1_000_000))

Note: The exact rate limit dimensions (requests per minute, tokens per minute, and any others), their specific numeric values, and how they vary by usage tier can change over time and differ by account. Confirm your account's actual current limits against your platform account's dashboard rather than assuming a specific number.

Knowing which dimension is actually binding matters for deciding what to fix: an application making many small, fast requests might be hitting a requests-per-minute limit well before a tokens-per-minute limit becomes relevant at all, while an application making fewer but much larger requests (a long document analyzed with Unit 7's input_file, for instance) might hit the tokens-per-minute limit first, even at a low request volume.

Reading Rate Limit Information From Response Headers

A response typically includes headers reporting the current rate limit status — how much of the limit has been used and how much remains before the next reset — which is more informative than waiting to actually hit a 429 error to find out.

response = client.responses.with_raw_response.create(model="gpt-5.6-terra", input="Hello")

remaining_requests = response.headers.get("x-ratelimit-remaining-requests")
remaining_tokens = response.headers.get("x-ratelimit-remaining-tokens")
print(f"Remaining requests: {remaining_requests}, remaining tokens: {remaining_tokens}")

Note: The exact header names, whether they're available through with_raw_response or a similar mechanism, and what they report can vary by SDK version. Confirm the current mechanism for reading rate limit headers against your installed SDK version's documentation.

Proactively checking how close a request came to the limit — rather than only reacting after receiving an actual 429 — lets an application throttle its own request rate ahead of time, smoothing out usage instead of bursting until it's rejected and only then backing off.

Handling a Genuine Spend Limit

A spend limit, unlike a rate limit, doesn't clear on its own — the account's configured spending cap has actually been reached, and every subsequent request fails identically until the limit is raised through the platform's own account settings.

def handle_response_error(exception, status_code: int) -> str:
    if status_code == 429:
        error_message = str(exception).lower()
        if "quota" in error_message or "billing" in error_message:
            return "spend_limit_reached"
        return "rate_limited"
    return "other_error"

Note: The exact wording used to distinguish a spend-limit error from an ordinary rate-limit error within a 429 response can vary by SDK version and by the specific error message returned. Confirm the current distinguishing detail against your installed SDK version's documentation, since both conditions can share the same status code.

Distinguishing these two cases inside application code — even though both surface as a 429 — is what prevents a spend-limit condition from being treated as a transient, retryable failure: retrying a request that's failing because of an exhausted budget wastes attempts on something that structurally cannot succeed until a person raises the limit or a new billing period begins, exactly the kind of client-error-versus-retryable-error distinction Lesson 1 introduced, applied to a more specific case.

Building Your Own Application-Level Rate Limiting

Rather than relying solely on the platform's own limits and reacting to a 429 when hit, an application serving many users can throttle its own request rate proactively, spreading requests out to stay comfortably under the limit in the first place.

import time
from collections import deque

class RequestRateLimiter:
    def __init__(self, max_requests_per_minute: int):
        self.max_requests_per_minute = max_requests_per_minute
        self.request_timestamps = deque()

    def wait_if_needed(self) -> None:
        now = time.time()
        while self.request_timestamps and now - self.request_timestamps[0] > 60:
            self.request_timestamps.popleft()

        if len(self.request_timestamps) >= self.max_requests_per_minute:
            sleep_time = 60 - (now - self.request_timestamps[0])
            if sleep_time > 0:
                time.sleep(sleep_time)

        self.request_timestamps.append(time.time())

limiter = RequestRateLimiter(max_requests_per_minute=100)

def call_with_self_throttling(client, input_text: str):
    limiter.wait_if_needed()
    return client.responses.create(model="gpt-5.6-terra", input=input_text)

RequestRateLimiter tracks the timestamps of recent requests within a rolling 60-second window and proactively pauses before making a new one if the recent count is already at the configured cap — a deliberately conservative application-level limit set somewhat below the platform's actual limit gives a safety margin, so a burst of legitimate traffic doesn't push the application straight into a real 429 from the platform.

Handling Rate Limits in a System With Multiple Independent Callers

An application serving multiple users concurrently — a web server handling requests from many different people at once — needs its rate limiting to be shared across every concurrent caller, not tracked independently per request, since the platform's limit applies to the account as a whole, regardless of how many separate parts of the application are making requests.

import threading

class ThreadSafeRateLimiter:
    def __init__(self, max_requests_per_minute: int):
        self.max_requests_per_minute = max_requests_per_minute
        self.request_timestamps = deque()
        self.lock = threading.Lock()

    def wait_if_needed(self) -> None:
        with self.lock:
            now = time.time()
            while self.request_timestamps and now - self.request_timestamps[0] > 60:
                self.request_timestamps.popleft()
            if len(self.request_timestamps) >= self.max_requests_per_minute:
                sleep_time = 60 - (now - self.request_timestamps[0])
                if sleep_time > 0:
                    time.sleep(sleep_time)
            self.request_timestamps.append(time.time())

Adding a threading.Lock here ensures that concurrent requests from different threads all check and update the same shared count safely, rather than each thread tracking its own independent view of recent request volume and collectively exceeding the actual account-wide limit despite each individual thread believing it was staying under its own tracked limit.

Common Mistakes

Treating every 429 error identically, retrying a genuine spend-limit error the same way as an ordinary rate-limit error, when a spend limit will never clear on its own no matter how long the retry loop waits.

Reacting to rate limits only after hitting a 429, rather than proactively checking rate limit headers or self-throttling to stay comfortably under the limit in the first place.

Tracking rate limiting independently per request or per thread in a concurrent application, rather than sharing a single rate-limit tracker across every caller, and consequently exceeding the actual account-wide limit despite each individual tracker believing it was within bounds.

Setting an application's self-imposed rate limit exactly at the platform's actual limit, leaving no safety margin for a burst of legitimate traffic to push the application into an actual 429.

Best Practices

Distinguish a genuine spend-limit condition from an ordinary rate limit in application code, even though both can surface as the same 429 status code, since only one of them is meaningfully retryable.

Check rate limit response headers proactively rather than waiting to hit an actual 429, allowing an application to throttle itself ahead of time.

Share rate-limiting state across every concurrent caller in a multi-threaded or multi-request application, rather than tracking it independently per thread or per request.

Set a self-imposed application rate limit somewhat below the platform's actual limit, leaving a safety margin for bursts of legitimate traffic.