Cache Best Practices

Bringing the Section Together

Lessons 121–129 covered how caching works, how to enable it, how to read its results, and how to test it. This final lesson collects the practices that hold across all of them, and adds three techniques that do not fit elsewhere: pre-warming the cache, sharing a cache across parallel work, and planning caching for batch jobs.

The core principles are short:

1. Cache what repeats. A prefix needs reuse to pay back its write.
2. Keep the prefix identical. Any change before a breakpoint causes a miss.
3. Place breakpoints at stable boundaries. The end of each section that changes at a different rate.
4. Match the lifetime to the traffic. The default 5 minutes for frequent use, the 1-hour TTL for spaced use.
5. Measure. The usage fields show whether the principles are holding.

Pre-Warming the Cache

A user's first request after a cold start pays the write premium and the full processing time for the prefix. Pre-warming writes the prefix before any user arrives, so the first real request reads it instead.

According to the documentation, pre-warming uses a request with max_tokens set to 0. The API processes the prompt, writes the cache at each breakpoint, and returns immediately with no output and no billed output tokens. The response has an empty content list and stop_reason of "max_tokens", which is expected here.

Three details matter:

• Put the breakpoint on the shared block, such as the system prompt or tools, not on the placeholder message. Otherwise the entry is keyed to the placeholder and the real requests miss it.
• Use the same model, thinking configuration, and effort setting as the real requests. Those values are part of the cached prefix.
• Refresh the cache before it expires. A 5-minute entry needs a new pre-warm request at least every 5 minutes, or a 1-hour entry every hour.

The documentation also lists requests that max_tokens: 0 does not accept: streaming, extended thinking enabled, structured outputs, and forced tool choice. It is also rejected inside a batch request.

"""prewarm.py: write the shared prefix before users arrive, then serve real requests."""

from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-opus-5-5"
SYSTEM = [{"type": "text", "text": "You are the GOCOURSE support assistant. " * 80,
           "cache_control": {"type": "ephemeral"}}]


def prewarm(system_blocks: list[dict], tools: list[dict] | None = None) -> int:
    """Write the cache without generating output. Returns the number of tokens written."""
    request = {
        "model": MODEL,
        "max_tokens": 0,
        "system": system_blocks,
        "messages": [{"role": "user", "content": "warmup"}],
    }
    if tools:                                    # Omit the key entirely when there are no tools
        request["tools"] = tools
    response = client.messages.create(**request)
    return response.usage.cache_creation_input_tokens


if __name__ == "__main__":
    print("written by pre-warm:", prewarm(SYSTEM))
    message = client.messages.create(
        model=MODEL,
        max_tokens=200,
        system=SYSTEM,
        messages=[{"role": "user", "content": "How do I get my certificate?"}],
    )
    print("real request read:", message.usage.cache_read_input_tokens)

Expected output (numbers vary; the pattern is the point):

written by pre-warm: <large number>
real request read: <the same large number>

How It Works

1. max_tokens: 0 tells the API to process the prompt and write the cache, without generating any output.
2. The placeholder message is required because a request needs a message, but it is never answered. The breakpoint sits on the system block, so the entry covers the shared prefix and not the placeholder.
3. The tools key is added only when tools exist. Passing tools=None may be rejected, so the code leaves the key out instead.
4. The real request uses the same system block, so its prefix matches the pre-warmed entry.

Parallel Requests and the First Write

A cache entry becomes available only after the first response starts. If you send many requests at the same moment, they all arrive before any entry exists, so all of them write. The documentation advises waiting for the first response to start, then sending the rest:

def fan_out(first_request, other_requests):
    first = client.messages.create(**first_request)          # Writes the entry
    with ThreadPoolExecutor(max_workers=4) as pool:           # The rest read it
        others = list(pool.map(lambda r: client.messages.create(**r), other_requests))
    return [first, *others]

The sequential first call is the key. In a web application, the same idea applies to user traffic: the first request after a cold start warms the cache for the rest.

Caching for Batch Jobs

The Message Batches API (Lessons 131–140) processes requests asynchronously, and its cache hits are best effort: requests are processed concurrently and in any order, so a shared prefix may not be read by the requests that follow. For batches, the documentation suggests a specific pattern:

1. Gather the batch requests that share a prefix.
2. Send one request containing that prefix, marked with the 1-hour TTL. This writes the entry.
3. Once it completes, submit the rest of the batch.
4. Monitor the job to know when the first request has finished.

The 1-hour TTL is recommended here because batches often take between 5 minutes and an hour to complete, which is longer than the default lifetime.

Isolation and Data Handling

Caches are isolated between organizations, and on the Claude API, Claude Platform on AWS, and Microsoft Foundry, also between workspaces within an organization. Identical prompts in different organizations never share a cache. Caching is eligible for zero data retention: the API keeps the cached representations in memory for the cache lifetime, not as stored text.

Two practical consequences follow. Workspaces are separate caches, so a team that uses several workspaces should expect separate warm-up for each. And the cache is not a store of your data. Deleting a prompt from your own system does not depend on the cache, and entries expire on their own.

What Can and Cannot Be Cached

Reviewing a Cache Plan

Before a feature ships, review its requests against the rules from this section. The review function below combines the structural checks from Lesson 128 with the rules that apply to specific block types:

"""cache_review.py: review a request's caching plan before it ships."""

from cache_checks import check_breakpoints, ordered_blocks


def review_cache_plan(request: dict) -> list[str]:
    problems = check_breakpoints(request)
    has_marker = False
    for location, block in ordered_blocks(request):
        if "cache_control" not in block:
            continue
        has_marker = True
        if block.get("type") == "thinking":
            problems.append(f"{location}: thinking blocks cannot be marked directly")
        if block.get("type") == "text" and not block.get("text", "").strip():
            problems.append(f"{location}: empty text blocks cannot be cached")
        if block.get("citations"):
            problems.append(f"{location}: mark the top-level document, not its citations")
    if not has_marker and "cache_control" not in request:
        problems.append("no cache markers: nothing will be cached")
    return problems


if __name__ == "__main__":
    request = {
        "model": "claude-opus-5-5",
        "max_tokens": 1024,
        "system": [{"type": "text", "text": "You are a tutor.", "cache_control": {"type": "ephemeral"}}],
        "messages": [
            {"role": "user", "content": "Explain closures."},
            {"role": "assistant", "content": [
                {"type": "thinking", "thinking": "...", "signature": "sig",
                 "cache_control": {"type": "ephemeral"}},
                {"type": "text", "text": "A closure is a function that remembers its environment."},
            ]},
            {"role": "user", "content": [
                {"type": "text", "text": "   ", "cache_control": {"type": "ephemeral"}},
            ]},
        ],
    }
    print(review_cache_plan(request) or "no problems")

Output:

['messages[1].content[0]: thinking blocks cannot be marked directly', 'messages[2].content[0]: empty text blocks cannot be cached']

How It Works

1. check_breakpoints() runs first, so the count and TTL rules are covered without repeating them.
2. Block-level rules catch markers that the API will not honor. A thinking block cannot carry a marker, and an empty text block is skipped. Both problems are reported with their location, so the fix is easy to find.
3. The citation rule flags a marker placed on a citation, where the documentation says the top-level document should be marked instead.
4. The missing-marker check reports a plan with no markers at all, which is a common result of a refactor that drops the caching code.
5. The review runs offline, so it can be part of code review, testing, or startup checks.

Common Mistakes

Caching Without Measuring

What happens: the team believes the feature is cached, but the usage fields show no reads for weeks.

Fix: log the fields, and alert on a falling hit rate.

Forgetting the Write Premium

What happens: features with one request per prefix cost more with caching than without.

Fix: estimate reads per prefix before enabling caching (Lesson 126).

Pre-Warming the Wrong Block

What happens: the pre-warm writes an entry that the real requests never match.

Fix: put the breakpoint on the shared block, and match the configuration of the real requests.

Parallel First Requests

What happens: a burst of traffic writes many entries, and no reads happen during the burst.

Fix: send one request first, then the rest.

Batch Jobs Without a Warm-Up

What happens: the batch writes the prefix many times, because its requests run concurrently.

Fix: send one 1-hour request first, then the rest.

Best Practices

• Cache repeated prefixes only, and estimate reads before enabling.
• Keep prefixes identical, and place breakpoints at stable boundaries.
• Match the TTL to request spacing.
• Pre-warm shared prefixes before traffic arrives, and refresh them before they expire.
• Send a first request before parallel traffic, and use the 1-hour TTL for batches.
• Review every caching plan with automated checks.
• Measure hit rates continuously, and investigate drops (Lessons 125 and 127).

Exercise

1. Change the placeholder message in prewarm.py to a real question, and explain why the pre-warm no longer serves its purpose.
2. Add a check to review_cache_plan() that flags a request with more than one marker on the same block.
3. Explain why prewarm() cannot use stream=True, and what the documentation says about that restriction.
4. Write a test that a plan with the marker on the placeholder message (the last user message) is flagged, and explain why the check is needed even though the API accepts the request.

Step 1: a real question would produce output, which the pre-warm is designed not to generate, and it would be billed as output tokens. Worse, if the breakpoint were on the question, the cache entry would be keyed to that specific question, so the real requests (with different questions) would never match it. The placeholder exists only to satisfy the requirement for a message, and it must be neither answered nor part of the shared prefix.

Step 4: the test builds a request whose only marker is on the final user message, and asserts that review_cache_plan() reports that the marker is on the varying part. The check is needed because the API does not reject such a request. It accepts it, writes an entry, and returns a valid response, so the plan appears to work. The entry is simply keyed to content that changes, so the savings never arrive, and the problem is visible only in the usage fields. A structural check catches it before the cost does.