Use Cache Breakpoints

Several Sections, Several Breakpoints

A single breakpoint caches everything before it. That is enough when the whole prefix changes at the same rate. Many real requests have sections that change at different rates:

With one breakpoint at the end, a change to any section invalidates everything after it. With a breakpoint at the end of each section, a change to one section invalidates only that section and the ones after it. The API allows up to four breakpoints per request, so each section can be cached independently.

The Layout

The cache order is fixed: tools, then system, then messages. A breakpoint caches everything up to and including itself in that order. Place breakpoints at the end of each section:

┌──────────────────────────────────────┐
│ tools (last tool marked)             │  breakpoint 1: changes rarely
├──────────────────────────────────────┤
│ system: instructions (marked)        │  breakpoint 2: changes rarely
├──────────────────────────────────────┤
│ system: retrieved documents (marked) │  breakpoint 3: changes per conversation
├──────────────────────────────────────┤
│ messages: history ... final user turn│  breakpoint 4: grows every turn
│ (marked)                             │
└──────────────────────────────────────┘

Each breakpoint writes its own entry. When a new turn is added, the final breakpoint moves forward, and the earlier entries are still read.

Building the Request

A function that assembles the request in this order keeps the placement consistent, and makes it easy to check:

"""breakpoints.py: place up to four cache breakpoints for a retrieval-augmented chat."""

from cache_checks import check_breakpoints, ordered_blocks

CACHE = {"type": "ephemeral"}


def render_documents(documents: list[tuple[str, str]]) -> str:
    return "\n\n".join(f"<document title={title!r}>\n{text}\n</document>" for title, text in documents)


def build_request(tools: list[dict], instructions: str, documents: list[tuple[str, str]],
                  history: list[dict], question: str, documents_stable: bool = True) -> dict:
    tools = [dict(tool) for tool in tools]
    if tools:
        tools[-1]["cache_control"] = CACHE                             # breakpoint 1: tool definitions

    system = [{"type": "text", "text": instructions, "cache_control": CACHE}]   # breakpoint 2
    doc_block = {"type": "text", "text": render_documents(documents)}
    if documents_stable:
        doc_block["cache_control"] = CACHE                             # breakpoint 3: only if reused
    system.append(doc_block)

    messages = [{"role": m["role"], "content": m["content"]} for m in history]
    messages.append({"role": "user", "content": [
        {"type": "text", "text": question, "cache_control": CACHE},   # breakpoint 4: the growing turn
    ]})
    return {"model": "claude-opus-5-5", "max_tokens": 1024,
            "tools": tools, "system": system, "messages": messages}


if __name__ == "__main__":
    tools = [
        {"name": "search_documents", "description": "Search the knowledge base.",
         "input_schema": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}},
        {"name": "get_document", "description": "Retrieve a document by ID.",
         "input_schema": {"type": "object", "properties": {"doc_id": {"type": "string"}}, "required": ["doc_id"]}},
    ]
    history = [
        {"role": "user", "content": "What is the refund window?"},
        {"role": "assistant", "content": "Refunds are available within 14 days of purchase."},
    ]
    documents = [("refund-policy", "Refunds: 14 days, under 30% completed.")]

    request = build_request(tools, "You are the GOCOURSE support assistant.", documents,
                            history, "What about certificates?")

    print("breakpoints:")
    for location, block in ordered_blocks(request):
        if "cache_control" in block:
            print(f"  {location}")
    print("check:", check_breakpoints(request) or "no problems")

    # Automatic caching uses one slot, so it cannot be combined with four explicit breakpoints
    with_automatic = dict(request, cache_control=CACHE)
    print("automatic + 4 explicit:", check_breakpoints(with_automatic))

Output:

breakpoints:
  tools[1]
  system[0]
  system[1]
  messages[2].content[0]
check: no problems
automatic + 4 explicit: ['5 breakpoints; the limit is 4']

How It Works

1. Breakpoint 1 goes on the last tool. Because all tools are cached as one prefix, marking the last one covers the whole list.
2. Breakpoint 2 goes on the stable instructions. A change to the documents does not affect this entry, because it comes before them.
3. Breakpoint 3 goes on the retrieved documents, but only when documents_stable is true. If the documents change on every question, a breakpoint there would write a new entry each time, so the flag keeps the marker off.
4. Breakpoint 4 goes on the final user turn. On the next turn, that entry becomes the earlier prefix, and it is read from the cache.
5. check_breakpoints() from Lesson 128 confirms the count and the TTL order.
6. The automatic marker uses a slot, so adding it to four explicit breakpoints produces five and is rejected.

Note: the history in step 4 is cached only when it stays identical. The assistant's earlier turns are copied from the real responses (Lesson 122), so the prefix matches exactly on later requests.

Mixing TTLs

Sections that change at different rates can also use different lifetimes. Tools and instructions may be stable for days, so the 1-hour TTL suits them, while documents that change every few minutes can use the default. The rule from Lesson 128 applies: longer TTLs must come before shorter ones.

tools[-1]["cache_control"] = {"type": "ephemeral", "ttl": "1h"}      # stable: longest TTL first
system[0]["cache_control"] = {"type": "ephemeral", "ttl": "1h"}      # stable: longest TTL first
system[1]["cache_control"] = {"type": "ephemeral"}                   # changes often: 5 minutes

The documentation describes the billing for mixed TTLs in detail. The practical rule is simple: put the stable sections first, give them the longer TTL, and check the order with check_breakpoints().

The Twenty-Block Lookback

A breakpoint can read an entry written by an earlier request only if that entry is within 20 blocks of the breakpoint. The documentation describes this window as part of how the system finds the longest matching prefix.

In a growing conversation, this limit matters. Suppose the final breakpoint is on block 35, and the last entry was written on block 15. The window covers blocks 35 down to 16, so the entry at block 15 is outside it, and there is no hit. The fix is to add a second breakpoint earlier, close to where the previous entry was written, so that a write accumulates there before it is needed. Breakpoint 4 in the layout above is designed for this: each turn's final user block is marked, so the previous turn's entry is always close.

Common Mistakes

Using One Breakpoint for Everything

What happens: a change to a document invalidates the instructions and the tools as well.

Fix: place breakpoints at the end of each section that changes at a different rate.

Marking Sections That Change Every Request

What happens: each request writes new entries, and the write premium is paid for nothing.

Fix: mark only sections that are reused, and leave changing sections unmarked.

Exceeding the Limit with Automatic Caching

What happens: the API rejects the request, or the automatic breakpoint finds no slot.

Fix: count the automatic marker in the total, and check with check_breakpoints().

Putting Shorter TTLs First

What happens: the API rejects the request, because longer TTLs must come first.

Fix: order the sections from most stable to least stable.

Ignoring the Lookback in Long Conversations

What happens: caching works for the first few turns, then stops.

Fix: keep the final breakpoint on each turn, and add an earlier breakpoint when the conversation grows beyond the window.

Best Practices

• Use one breakpoint per section that changes at its own rate.
• Use all four slots deliberately, leaving room for automatic caching only when you need it.
• Mark a section only when it is reused.
• Order sections from most to least stable, with longer TTLs first.
• Check every request with check_breakpoints() in tests (Lesson 128).
• Watch long conversations for the lookback limit.

Exercise

1. Set documents_stable=False and print the breakpoints. How many remain, and why does the check still pass?
2. Add a third tool, and confirm that the marker moves to tools[2], not to the previous tool.
3. Build a request with a 1-hour breakpoint on the documents and a 5-minute breakpoint on the instructions. What does check_breakpoints() report, and how do you fix it?
4. Explain why breakpoint 4 is placed on the final user turn rather than on the assistant's last reply.

Step 1: three breakpoints remain: the last tool, the instructions, and the final user turn. The check passes, because three is within the limit of four. The documents block is unmarked, so its content is processed as ordinary input on each request. That is the right trade-off when the documents change every question, since the write premium would otherwise be paid each time.

Step 4: the final user turn is the point where the conversation will continue. On the next request, the previous turns form the prefix, and the marker on this turn writes the entry that the next request can read. Marking the assistant's reply would place the boundary one turn later than needed, because the reply is always followed by a new user message. Marking the user turn means the entry covers everything up to the point where the next user message begins.