Debug Cache Misses

When Caching Does Not Happen

A feature that should be cheap shows cache_read_input_tokens: 0 on every request. Nothing fails, and nothing reports an error. Caching is silent by design: a prompt that is too short, or changed in the wrong place, is processed normally and billed at the full price (Lesson 121).

Debugging a miss means answering two questions:

1. Did the request qualify? Was the prefix long enough, and was the cache marker placed correctly?
2. Did the prefix match? What changed between the request that wrote the cache and the one that failed to read it?

The Common Causes

Most misses come from a short list of causes, and the documentation describes each:

The documentation's table of invalidations shows how far each change reaches. A change to tool definitions invalidates the tools, system, and message caches. A change to tool choice, or adding an image anywhere in the prompt, invalidates only the messages. The later a change occurs in the prompt, the less it costs.

Finding the Divergence

When a prefix that should match does not, the fastest way to find the cause is to compare the two requests block by block. The first block that differs is the cause, and if it lies before the cache marker, it explains the miss.

The debugging module below turns each request into an ordered list of block fingerprints, in the same order the cache uses (tools, then system, then messages), and reports the first difference:

"""cache_debug.py: find the first block where two requests diverge, relative to cache markers."""

import hashlib
import json


def digest(obj) -> str:
    return hashlib.sha256(json.dumps(obj, sort_keys=True, ensure_ascii=False).encode("utf-8")).hexdigest()[:12]


def prefix_blocks(request: dict) -> list[tuple[str, str, bool]]:
    """Blocks in cache order: (location, digest, has_cache_marker)."""
    blocks = []
    for i, tool in enumerate(request.get("tools", [])):
        blocks.append((f"tools[{i}] {tool.get('name')}", digest(tool), "cache_control" in tool))
    system = request.get("system", [])
    if isinstance(system, str):
        system = [{"type": "text", "text": system}]
    for i, block in enumerate(system):
        blocks.append((f"system[{i}]", digest(block), "cache_control" in block))
    for m, message in enumerate(request.get("messages", [])):
        content = message["content"]
        if isinstance(content, str):
            content = [{"type": "text", "text": content}]
        for i, block in enumerate(content):
            location = f"messages[{m}].content[{i}] ({message['role']})"
            blocks.append((location, digest(block), "cache_control" in block))
    return blocks


def explain(first: dict, second: dict) -> str:
    """Explain whether the second request can read the cache written by the first."""
    a, b = prefix_blocks(first), prefix_blocks(second)
    markers = [index for index, (_, _, marked) in enumerate(a) if marked]
    if not markers:
        return "no cache marker in the first request: nothing is cached"
    last_marker = markers[-1]

    for index, (block_a, block_b) in enumerate(zip(a, b)):
        if block_a[1] != block_b[1]:
            if index <= last_marker:
                return f"MISS: block {index} ({block_a[0]}) differs and lies inside the cached prefix"
            return f"HIT: block {index} ({block_a[0]}) differs but lies after the last breakpoint"

    if len(a) != len(b):
        shorter = min(len(a), len(b))
        if shorter > last_marker:
            return "HIT: the second request is longer, and its prefix matches up to the last breakpoint"
        return "MISS: the requests have different numbers of blocks inside the cached prefix"
    return "identical prefixes: the cache should hit"


if __name__ == "__main__":
    tools = [{"name": "get_weather", "description": "Get the weather for a city.",
              "input_schema": {"type": "object", "properties": {"city": {"type": "string"}}}}]
    static_text = "You are a support assistant for GOCOURSE. Answer briefly."

    def request(system_blocks, tool_list=tools):
        return {"model": "claude-opus-5-5", "tools": tool_list, "system": system_blocks,
                "messages": [{"role": "user", "content": "Where is my certificate?"}]}

    marked = {"type": "text", "text": static_text, "cache_control": {"type": "ephemeral"}}

    # Case 1: the timestamp lives in an unmarked block after the breakpoint
    a = request([marked, {"type": "text", "text": "Time: 10:00"}])
    b = request([marked, {"type": "text", "text": "Time: 10:05"}])
    print(explain(a, b))

    # Case 2: the timestamp lives inside the marked block
    a = request([{"type": "text", "text": f"{static_text} Time: 10:00",
                  "cache_control": {"type": "ephemeral"}}])
    b = request([{"type": "text", "text": f"{static_text} Time: 10:05",
                  "cache_control": {"type": "ephemeral"}}])
    print(explain(a, b))

    # Case 3: a tool description changed, which sits before the marker
    changed_tools = [{**tools[0], "description": "Get the current weather for a city."}]
    a = request([marked])
    b = request([marked], tool_list=changed_tools)
    print(explain(a, b))

Output:

HIT: block 2 (system[1]) differs but lies after the last breakpoint
MISS: block 1 (system[0]) differs and lies inside the cached prefix
MISS: block 0 (tools[0] get_weather) differs and lies inside the cached prefix

The block numbers count the tool definition as block 0, because tools come first in the cache order. That is why the system blocks start at 1, not 0.

How It Works

1. digest() fingerprints each block by hashing its content, with keys sorted so that dictionary order does not matter for the comparison.
2. prefix_blocks() lists the blocks in the order the cache uses: tools, then system, then messages. Each entry records whether the block carries a marker, which tells the checker where the breakpoints are.
3. explain() finds the first block that differs. Only a difference at or before the last marker can prevent a hit. A difference after it cannot affect what has already been cached.
4. Case 1 changes the timestamp in an unmarked block after the breakpoint. The tool is block 0, the marked system block is block 1, and the changed timestamp is block 2. The cached prefix still matches, so the explanation is a hit, which is the correct design.
5. Case 2 puts the same timestamp inside the marked block. The changed block is block 1, which is at the marker, so the explanation is a miss.
6. Case 3 changes a tool description. Tools come first in the cache order, so even an unchanged system prompt cannot save the cache.

Note: the checker compares the content of each block, which is the right model for explaining misses. The API compares the exact bytes it receives, so a fingerprint can miss differences that matter at the byte level. The documentation notes that key ordering in tool-use inputs can break caches in some languages. Compare the serialized request you actually send, not only the Python objects, when a miss remains unexplained.

Diagnosing a Real Miss

Use the checker with the requests your application actually sends:

1. Log the request with its usage fields, and keep the last request that wrote the cache.
2. Run explain() between that request and the one that missed.
3. Read the first difference: a change inside the prefix explains the miss, and a change after the marker does not.
4. Check the rules that the checker does not cover: minimum length, time since the last request, parallel first requests, and changes to tool choice, images, or thinking settings.

Common Mistakes

Assuming a Miss Is a Bug in the API

What happens: time is spent on the platform when the cause is a timestamp in the prefix.

Fix: compare the requests, and check the first difference.

Debugging Only the Last Request

What happens: the request that matters is the one that wrote the entry, not the one that failed to read it.

Fix: keep the writing request's content, and compare it with the miss.

Fixing the Symptom Instead of the Position

What happens: the marker is moved, but the changing content stays inside the prefix.

Fix: move the changing content after the marker.

Overlooking Settings Outside the Messages

What happens: a change to the thinking configuration or effort setting silently invalidates the messages cache.

Fix: keep these settings constant across requests that should share a cache.

Best Practices

• Log cache_read_input_tokens and cache_creation_input_tokens for every request.
• Keep a copy of the request that wrote each entry, at least for the most common prefixes.
• Compare requests block by block before changing code.
• Keep changing content after the last breakpoint.
• Treat sudden hit-rate drops as a signal to diff the latest deployment.
• Build cached text once, and avoid per-request formatting of stable content.

Exercise

1. Add a second unmarked block after the timestamp block in Case 2, and check whether explain() changes its answer. Why or why not?
2. Change the tool's input_schema (not its description), and rerun Case 3. What does the checker report?
3. Make explain() also report the number of blocks inside the cached prefix, so the report shows how large the cache entry is.
4. Explain why digest() uses sort_keys=True, and what kind of problem this helps avoid, and what kind it cannot catch.

Step 1: no change in the outcome. The first difference is still at block 1, the marked block, which lies inside the cached prefix, so the answer is still a miss. A block after the first difference cannot change the explanation, because the miss has already been decided by the earlier change.

Step 2: the checker reports the same miss, but for a different reason at the same place: block 0 (tools[0] get_weather) differs and lies inside the cached prefix. Any change to a tool definition, including its input schema, is a change to the tools block, and tools come first in the cache order. The digest includes the whole tool dictionary, so schema changes are detected just like description changes.

Step 4: sort_keys=True makes two dictionaries with the same keys and values produce the same fingerprint, regardless of the order in which the keys were inserted. That avoids false alarms when two requests are built in different orders but mean the same thing. It cannot catch the real problem the API may see: if the serialized bytes differ in key order, the API may treat the requests as different, even though the checker reports a match. That is why the note recommends comparing the exact serialized request when a miss remains unexplained.