Set Usage Limits

Bounding What Can Go Wrong

Every defense in this section reduces the chance that something goes wrong. Usage limits accept that something may go wrong, and they bound the damage. A loop that never ends, a user who sends a thousand requests an hour, a prompt injection that makes the assistant summarize every document it can reach, or a key that leaks and is used by a stranger: each of these can produce a bill far larger than the application was designed to cost. A limit turns an unbounded loss into a bounded one, and it gives someone time to notice and respond.

Limits operate at several levels, from a single request to the whole account. The lessons in this section and earlier ones have introduced most of them. This lesson gathers them into one design, adds a per-user budget that enforces spending in code, and explains how alerts and account-level controls complete the picture.

Limits at Each Level

Each level of limit catches a different failure, and the most useful limits are layered:

The request and run limits are set in the code that makes calls. The user budget is application logic, and the workspace limit is an account setting. Alerts are the warning that tells a person a limit is about to matter. A good design uses all five, so that a failure at one level is caught by another.

Enforcing a Per-User Budget

A per-user budget is the limit most likely to be missing, and it is simple to build. Before each call, the application reserves an estimate of the tokens the call may use. After the call, it settles the reservation against the actual usage, releasing any unused part. A call is refused when its estimate would exceed the remaining budget. The reservation step matters because it prevents a burst of concurrent requests from each seeing the same remaining budget and all proceeding:

"""usage_budget.py: a per-user daily token budget, enforced with reservations."""

from dataclasses import dataclass, field


@dataclass
class UsageLedger:
    daily_budget: int
    used: dict[str, int] = field(default_factory=dict)

    def reserve(self, user: str, estimate: int) -> tuple[bool, int]:
        spent = self.used.get(user, 0)
        remaining = self.daily_budget - spent
        if estimate > remaining:
            return False, remaining
        self.used[user] = spent + estimate
        return True, remaining - estimate

    def settle(self, user: str, reserved: int, actual: int) -> None:
        self.used[user] = self.used.get(user, 0) + (actual - reserved)


if __name__ == "__main__":
    ledger = UsageLedger(daily_budget=1000)
    ok, remaining = ledger.reserve("student-42", 400)
    print("step 1:", ok, "remaining", remaining)
    ok, remaining = ledger.reserve("student-42", 400)
    print("step 2:", ok, "remaining", remaining)
    ok, remaining = ledger.reserve("student-42", 400)
    print("step 3:", ok, "remaining", remaining)
    ledger.settle("student-42", reserved=400, actual=250)
    print("step 4 settle: used", ledger.used["student-42"])
    ok, remaining = ledger.reserve("student-42", 400)
    print("step 5:", ok, "remaining", remaining)
    ok, remaining = ledger.reserve("student-42", 300)
    print("step 6:", ok, "remaining", remaining)

Output:

step 1: True remaining 600
step 2: True remaining 200
step 3: False remaining 200
step 4 settle: used 650
step 5: False remaining 350
step 6: True remaining 50

How It Works

1. The first two reservations succeed, each taking 400 from the budget of 1000. The remaining amount falls to 600, then 200.
2. The third reservation is refused because 400 exceeds the 200 that remain. The request is never sent, so no tokens are spent.
3. Settling corrects the reservation. The first call actually used 250 tokens, so 150 of its 400 are released, and the total used falls from 800 to 650. Estimates are conservative, and settling returns the surplus.
4. The fifth step is refused again, because 400 still exceeds the 350 that now remain.
5. The sixth step succeeds with a smaller estimate, and the remaining budget reaches 50. A budget is enforced by what remains, not by how many calls were made.

The ledger is held in memory here, so the demonstration is simple. A production system stores the ledger in a database and performs the reservation as a single atomic update, so that two requests cannot both succeed against the same remaining amount.

Choosing the Numbers

A budget should reflect what a user legitimately needs, plus margin, and it should be reviewed when usage patterns change. Estimates should be realistic: the token count of a request can be obtained with the counting endpoint (Lesson 158), and the reservation can use that figure plus the expected output. A reservation that is far too high refuses legitimate work, and one that is far too low lets a runaway through, so the settle step should be monitored to see how well the estimates match reality.

Account-Level Limits and Alerts

A per-user budget does not stop a shared key from being used by something outside the application. For that, the account's workspace can be given its own spending limit in the console, and the key can be scoped to that workspace (Lesson 171). When spend approaches the limit, an alert should reach a person before the limit is reached, so that a legitimate increase can be approved and an unexpected one can be stopped. Lesson 186 covers tracking costs in detail, and the alert thresholds belong with that tracking.

A spend limit that is reached stops the workspace's requests, and the error it produces is a rate or quota error of the kind covered in Lesson 142. The application should treat it as a condition to report, not as a temporary fault to retry.

Summary

Usage limits bound the damage from runaway loops, abusive users, injections, and leaked keys, and they work best in layers: per request, per run, per user, per workspace, and by alert. A per-user budget is enforced in code through reservations that are settled against actual usage, which prevents concurrent requests from overspending. Estimates should be realistic and monitored. Workspace spend limits and alerts provide the account-level backstop. The next lesson completes the security section by looking at the logs that show whether these protections are working.