Test Cache Performance

Caching Is Easy to Break Silently

Lessons 121–127 covered how caching behaves. In a real application, the risk is that a change to a prompt, a tool, or a template quietly moves a breakpoint, adds a timestamp to the cached part, or pushes the prefix below the minimum length. Nothing fails. The bill rises, and the team notices weeks later.

Caching therefore deserves tests, like any other behavior that affects cost (Lesson 39). This lesson uses three layers:

The first two run on every change. The third runs when you want evidence that the platform is doing what the structure implies.

Structural Checks

A request that violates the documented rules will not cache, or will be rejected. Catch those mistakes before sending. The checker below walks the request in cache order (tools, system, messages), and reports:

• more than four breakpoints, counting an automatic top-level marker as one,
• markers that are not ephemeral, and
• a 5-minute breakpoint that comes before a 1-hour breakpoint, which the documentation says is not allowed.

"""cache_checks.py: structural checks on a request's cache markers."""


def ordered_blocks(request: dict):
    """Yield (location, block) in cache order: tools, system, then messages."""
    for i, tool in enumerate(request.get("tools", [])):
        yield f"tools[{i}]", tool
    system = request.get("system", [])
    if isinstance(system, str):
        system = [{"type": "text", "text": system}]
    for i, block in enumerate(system):
        yield f"system[{i}]", block
    for m, message in enumerate(request.get("messages", [])):
        content = message["content"]
        if isinstance(content, str):
            content = [{"type": "text", "text": content}]
        for i, block in enumerate(content):
            yield f"messages[{m}].content[{i}]", block


def check_breakpoints(request: dict) -> list[str]:
    problems = []
    markers = [(location, block["cache_control"])
               for location, block in ordered_blocks(request) if "cache_control" in block]
    automatic = 1 if "cache_control" in request else 0

    if len(markers) + automatic > 4:
        problems.append(f"{len(markers) + automatic} breakpoints; the limit is 4")
    for location, marker in markers:
        if marker.get("type") != "ephemeral":
            problems.append(f"{location}: unsupported cache type {marker.get('type')!r}")

    ttls = [marker.get("ttl", "5m") for _, marker in markers]
    if "5m" in ttls and "1h" in ttls[ttls.index("5m"):]:
        problems.append("a 1-hour breakpoint comes after a 5-minute breakpoint; put longer TTLs first")
    return problems

These checks run in microseconds, so they can run in code review, in a pre-commit hook, or at application startup on a sample request.

Diagnostic Tests

The diagnostic functions from Lesson 127 deserve tests too, since teams rely on them when investigating misses. Each test describes one known change and its expected explanation:

"""tests/test_cache.py: structural and diagnostic tests for prompt caching. No API calls."""

from cache_checks import check_breakpoints
from cache_debug import explain

STATIC = "You are a support assistant for GOCOURSE. " * 40
MARKED = {"type": "text", "text": STATIC, "cache_control": {"type": "ephemeral"}}


def request(system_blocks, tools=None):
    body = {"model": "claude-opus-5-5", "system": system_blocks,
            "messages": [{"role": "user", "content": "Where is my certificate?"}]}
    if tools is not None:
        body["tools"] = tools
    return body


def test_valid_request_has_no_problems():
    assert check_breakpoints(request([MARKED])) == []


def test_too_many_breakpoints_is_reported():
    blocks = [{"type": "text", "text": f"part {n}", "cache_control": {"type": "ephemeral"}}
              for n in range(5)]
    assert any("limit is 4" in problem for problem in check_breakpoints(request(blocks)))


def test_longer_ttl_must_come_first():
    blocks = [
        {"type": "text", "text": "short-lived", "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": "long-lived", "cache_control": {"type": "ephemeral", "ttl": "1h"}},
    ]
    assert any("1-hour breakpoint comes after" in problem for problem in check_breakpoints(request(blocks)))


def test_change_after_breakpoint_is_a_hit():
    first = request([MARKED, {"type": "text", "text": "Time: 10:00"}])
    second = request([MARKED, {"type": "text", "text": "Time: 10:05"}])
    assert explain(first, second).startswith("HIT")


def test_change_inside_breakpoint_is_a_miss():
    first = request([{**MARKED, "text": STATIC + "Time: 10:00"}])
    second = request([{**MARKED, "text": STATIC + "Time: 10:05"}])
    assert explain(first, second).startswith("MISS")

Each test name states a rule, and each assertion checks the rule's consequence. When a future change breaks one of them, the name says what behavior was lost.

Live Cache Test

Structural tests cannot prove that the platform actually reads the cache. A live test sends the same request twice and checks the usage fields. Because the cache may already contain the prefix from an earlier run, the first call can either write or read, so the test accepts either, and requires a read on the second call:

"""tests/test_cache_live.py: confirms the cache is read on a repeated request. Calls the API."""

import os

import pytest
from anthropic import Anthropic

LONG_SYSTEM = "You are a support assistant for GOCOURSE. Answer in one sentence. " * 60


@pytest.mark.skipif(not os.getenv("RUN_LIVE_TESTS"), reason="calls the live API; set RUN_LIVE_TESTS=1")
def test_second_identical_request_reads_the_cache():
    client = Anthropic()
    request = dict(
        model="claude-opus-5-5",
        max_tokens=50,
        system=[{"type": "text", "text": LONG_SYSTEM, "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": "What is a certificate?"}],
    )
    first = client.messages.create(**request).usage
    second = client.messages.create(**request).usage

    assert first.cache_creation_input_tokens + first.cache_read_input_tokens > 0, \
        "the prefix was not cached at all; check the minimum length"
    assert second.cache_read_input_tokens > 0, "the second request did not read the cache"

The test does not hard-code token counts, which would change with the tokenizer. It checks the behavior that matters: the prefix is cached, and the second request reads it.

Note: the live test depends on timing. If the two calls are separated by more than the TTL, the second call writes again, and the test fails for a reason unrelated to the code. Keep the calls close together, and run the test when the cache is not in an unusual state.

Running the Tests

python -m pip install pytest
python -m pytest -q tests/test_cache.py
RUN_LIVE_TESTS=1 python -m pytest -q tests/test_cache_live.py

5 passed in 0.04s
1 passed in 3.87s

The first command runs the five offline tests. The second runs the live test, which passes when the cache behaves as expected.

How It Works

1. ordered_blocks() walks the request in the same order the cache uses. Checks that depend on position, such as "a 1-hour breakpoint must come before a 5-minute one", need this order.
2. check_breakpoints() counts markers, including an automatic top-level marker, which uses one of the four slots (Lesson 122). It also checks cache types and TTL ordering.
3. The TTL rule looks for a 5-minute marker followed by a 1-hour marker. The documentation requires longer TTLs to appear first, so this pattern is an error.
4. The diagnostic tests pin down the behavior of explain(): a change after the breakpoint is a hit, and a change inside it is a miss.
5. The live test avoids brittle numbers. It asserts that caching happened at all, and that the second request read it.
6. The offline tests need no key and no network, so they can run on every commit.

Common Mistakes

Testing Only Whether the Code Runs

What happens: a refactor moves a breakpoint, every test still passes, and caching stops working.

Fix: test the cache markers and their positions directly.

Hard-Coding Token Counts

What happens: tests fail when the tokenizer or the prompt changes, even though caching works.

Fix: assert behavior: reads happen, writes happen, positions are correct.

Running Live Tests in Every Build

What happens: slow, costly builds, and failures caused by cache timing.

Fix: keep the live test opt-in, and run it on a schedule or before a release.

Forgetting the Minimum Length in Test Fixtures

What happens: a test prompt is too short, nothing is cached, and the test fails for the wrong reason.

Fix: make the fixture long enough, and assert on the minimum-length failure explicitly if you want to test it.

Best Practices

• Write structural checks for every cache marker in production requests.
• Test each documented invalidation that matters to your application.
• Keep the live cache test small and opt-in.
• Assert behavior, not exact token counts.
• Run the offline tests on every change, the live test before releases.
• Review cache-related diffs carefully, since they affect cost even when they do not affect answers.

Exercise

1. Add a test that a request with an automatic top-level marker and four explicit markers is reported as having five breakpoints.
2. Add a check that warns when a block marked for caching is empty (the documentation says empty text blocks cannot be cached).
3. Write a test that a change to tools after a marked system block is reported as a miss, and explain why this differs from a change to an unmarked message.
4. Explain why the live test accepts either a write or a read on its first call.

Step 1:

def test_automatic_plus_explicit_counts_the_automatic_marker():
    blocks = [{"type": "text", "text": f"part {n}", "cache_control": {"type": "ephemeral"}}
              for n in range(4)]
    body = request(blocks)
    body["cache_control"] = {"type": "ephemeral"}
    assert any("5 breakpoints" in problem for problem in check_breakpoints(body))

The automatic marker uses one of the four slots, so four explicit markers plus the automatic one exceed the limit. The check counts the top-level marker, which is easy to forget when the automatic option is added later.

Step 4: whether the first call writes or reads depends on what happened before the test started. If an earlier run has just cached the same prefix, the first call reads it. If not, the first call writes it. Either way, the prefix is now cached, which is the behavior under test. Requiring a specific outcome for the first call would make the test depend on the state of the cache, rather than on the code.