Limit Output Tokens

Why Limit Output?

Output tokens are the most expensive part of a request (5× the input price on current models) and the main driver of latency (Lesson 45). Limiting output is therefore one of the most effective ways to control cost, speed, and predictability.

Lesson 13 introduced max_tokens, the hard ceiling on generated tokens. This lesson treats output limiting as a strategy with several tools, each controlling a different aspect:

The key principle: shape the output with the first four tools, and use max_tokens as a safety net, not as the main control.

Layer 1: Ask for Less

The cheapest output is output Claude never plans to write. Specific length instructions (Lesson 38) reduce the visible answer:

messages = [{"role": "user", "content": "In 2 sentences, what is a Python virtual environment?"}]

Also remove requests for content you do not need. "Explain, give examples, list alternatives, and summarize" produces far more output than "Explain in 2 sentences."

Layer 2: Reduce Thinking with effort

Thinking tokens are often a large share of output (Lesson 45). For simple tasks, lower effort reduces them:

reply = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    output_config={"effort": "low"},
    messages=[{"role": "user", "content": "Classify the sentiment of: 'Loved the course!'"}],
)

Layer 3: Constrain the Shape with Structured Outputs

When you need data, not prose, structured outputs (Lesson 36) limit the response to exactly the fields you define, with no explanations, introductions, or extra commentary:

from pydantic import BaseModel


class Sentiment(BaseModel):
    label: str
    confidence: float


result = client.messages.parse(
    model="claude-opus-5-5",
    max_tokens=1024,
    output_config={"effort": "low"},
    output_format=Sentiment,
    messages=[{"role": "user", "content": "Classify: 'Loved the course!'"}],
)

A JSON object with two fields is far shorter than a paragraph explaining the classification.

Layer 4: The Safety Net, max_tokens

max_tokens guarantees that no single response exceeds a known size, whatever the prompt or input. Set it from measured output sizes (Lesson 45): roughly 1.5–2× the p95 of output_tokens for the feature.

Because you pay only for generated tokens, a slightly generous limit costs nothing in the normal case and protects against unusual cases. A limit that is too tight causes truncation and retries, which costs more.

Maximum Values

The maximum max_tokens for current models is 128,000 with the Messages API. With the Message Batches API, supported models can go up to 300,000 with a beta header (Lesson 13). Very large limits require streaming, because the SDK raises an error for non-streaming requests it expects to take longer than about 10 minutes.

Layer 5: Stopping a Stream Early

With streaming (Lessons 51–60), your code receives text as it is generated and can stop reading whenever it has enough, for example when a preview reaches a character limit, or when a user clicks "Stop":

collected = ""
with client.messages.stream(
    model="claude-opus-5-5",
    max_tokens=2048,
    messages=[{"role": "user", "content": "List 50 Python project ideas."}],
) as stream:
    for text in stream.text_stream:
        collected += text
        if len(collected) >= 500:
            break          # Leaving the 'with' block closes the stream

Leaving the with block closes the connection, and generation for that request ends. Tokens generated before the stream was closed are still billed. Early cancellation limits cost for open-ended requests, but it is not free. Lesson 60 covers closing streams safely.

Layer 6: Application Budgets

Per-request limits do not cap total spend. Applications that serve many users usually add budgets:

Lesson 179 (Set Usage Limits) implements user budgets. The practical example below combines per-feature limits with a simple per-user budget.

Practical Example: Per-Feature Output Profiles with a User Budget

This module defines an output profile for each feature (length instruction, effort, and max_tokens), applies it to every request, records output usage per user, and refuses requests that would exceed the user's daily output budget.

"""output_limits.py: per-feature output profiles and a simple per-user output budget."""

from collections import defaultdict
from dataclasses import dataclass

from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-opus-5-5"


@dataclass(frozen=True)
class OutputProfile:
    instruction: str
    effort: str
    max_tokens: int


PROFILES = {
    "classify": OutputProfile("Reply with only the label.", "low", 1024),
    "quick_answer": OutputProfile("Answer in at most 3 sentences.", "low", 1024),
    "explain": OutputProfile("Explain in about 200 words with one short code example.", "medium", 3072),
}

DAILY_OUTPUT_BUDGET = 4_000          # Output tokens per user per day (small, for demonstration)
usage_today: dict[str, int] = defaultdict(int)


class BudgetExceeded(Exception):
    pass


def ask(user_id: str, feature: str, question: str) -> str:
    profile = PROFILES[feature]

    if usage_today[user_id] + profile.max_tokens > DAILY_OUTPUT_BUDGET:
        raise BudgetExceeded(f"{user_id} has used {usage_today[user_id]} of "
                             f"{DAILY_OUTPUT_BUDGET} output tokens today.")

    reply = client.messages.create(
        model=MODEL,
        max_tokens=profile.max_tokens,
        output_config={"effort": profile.effort},
        messages=[{"role": "user", "content": f"{question}\n\n{profile.instruction}"}],
    )
    usage_today[user_id] += reply.usage.output_tokens

    text = "".join(b.text for b in reply.content if b.type == "text").strip()
    if reply.stop_reason == "max_tokens":
        text += "\n[Answer cut off at the feature's output limit.]"
    return text


if __name__ == "__main__":
    requests = [
        ("student-7", "classify", "Sentiment of: 'The quiz was confusing.' (positive/negative/neutral)"),
        ("student-7", "quick_answer", "What does pip do?"),
        ("student-7", "explain", "How do Python decorators work?"),
        ("student-7", "explain", "How do generators work?"),
    ]
    for user_id, feature, question in requests:
        try:
            answer = ask(user_id, feature, question)
            print(f"[{feature}] {answer[:70]}...")
        except BudgetExceeded as error:
            print(f"[{feature}] refused: {error}")
        print(f"   {user_id} output used today: {usage_today[user_id]}")

Example output (numbers vary):

[classify] negative...
   student-7 output used today: 96
[quick_answer] pip is Python's package installer. It downloads packages from PyPI and ins...
   student-7 output used today: 241
[explain] A decorator is a function that takes another function and returns a modi...
   student-7 output used today: 1012
[explain] refused: student-7 has used 1012 of 4000 output tokens today.
   student-7 output used today: 1012

How It Works

1. OutputProfile bundles the three output controls for a feature: a length instruction, an effort level, and a max_tokens ceiling. Each feature gets limits that fit its purpose, and a classification never receives the budget of a long explanation.
2. The instruction is appended to the question, so the visible answer is shaped by the prompt, the most efficient control.
3. The budget check reserves the worst case before sending. A request is refused if the user's usage plus the feature's max_tokens would exceed the budget. In the sample run, the fourth request is refused: 1,012 tokens already used plus the 3,072 tokens reserved for an explain request is 4,084, which exceeds the 4,000-token budget. The third request was allowed, because 241 + 3,072 = 3,313 fits within the budget.
4. Actual usage is recorded after the reply, from usage.output_tokens, which includes thinking. Usage grows by what was really generated, not by the reservation.
5. Truncation is reported to the user, so a cut-off answer is never presented as complete.

Note: usage_today lives in memory and resets when the program restarts. Production systems store usage in a database or cache, keyed by user and date (Lesson 179).

Common Mistakes

Relying on max_tokens Alone

What happens: answers are cut off mid-sentence instead of being concise, and thinking-heavy requests return no text.

Fix: shape length with instructions and effort; use max_tokens as the safety net.

One Limit for Every Feature

What happens: simple features waste budget, and complex features are truncated.

Fix: define output profiles per feature, based on measurements.

Forgetting That Thinking Counts

What happens: limits sized for the visible answer cause truncation.

Fix: size limits from measured output_tokens, and lower effort where thinking is not needed.

Assuming Early Stream Cancellation Is Free

What happens: costs are underestimated for cancelled requests.

Fix: tokens generated before cancellation are billed. Use cancellation to cap cost, not to eliminate it.

No Total Budget

What happens: per-request limits are respected, but a single user or script sends thousands of requests.

Fix: add per-user, per-feature, and organization-level budgets.

Best Practices

• Shape output with instructions first, then effort, then structured outputs.
• Use max_tokens as a measured safety net (1.5–2× p95).
• Define output profiles per feature, and keep them in configuration.
• Reserve the worst case when checking budgets, and record actual usage afterward.
• Report truncation to users and in metrics.
• Use streaming for long outputs, and let users stop generation they no longer need.
• Re-measure after changing models, prompts, or effort.

Exercise

1. Change DAILY_OUTPUT_BUDGET to 5_000. Before running the script, predict whether the fourth request is refused, then run it to check.
2. Add a "summary" profile for summarizing documents in about 100 words, with suitable effort and max_tokens values.
3. Change the budget check to use the feature's average measured output instead of max_tokens. What are the advantages and risks?
4. Modify the streaming example to stop after the first 5 complete lines instead of 500 characters.

Step 1: with a budget of 5,000, the check before the fourth request is 1,012 + 3,072 = 4,084, which does not exceed 5,000, so the request is allowed. (Your exact usage numbers may differ slightly, because output sizes vary between runs.)

Step 3: using the average reserves less, so users can make more requests before reaching the limit. The risk is that a request larger than average can push usage over the budget, because the check no longer covers the worst case. A common compromise is to reserve the p95 instead of the average or the maximum.

Step 4:

collected = ""
with client.messages.stream(
    model="claude-opus-5-5",
    max_tokens=2048,
    messages=[{"role": "user", "content": "List 50 Python project ideas, one per line."}],
) as stream:
    for text in stream.text_stream:
        collected += text
        if collected.count("\n") >= 5:
            break

print("\n".join(collected.splitlines()[:5]))

Counting newline characters detects when five lines are complete. The final splitlines()[:5] removes any partial sixth line.