Why the Same Prefix Keeps Getting Billed

A great many real applications send requests that share a large, unchanging prefix from one call to the next — Unit 11's agents send the same system instructions on every single turn of a run; Unit 9's file-search-enabled assistant resends the same lengthy tool definitions with every call; a customer support application built on Unit 6's structured outputs sends the same detailed formatting instructions on every request, with only the customer's specific message actually changing. Without any special handling, the platform reprocesses that entire shared prefix from scratch on every request, even though its content — and therefore the computation needed to process it — is identical to the previous call. Prompt caching is the mechanism that avoids repeating this redundant work, and understanding it changes how you should structure a prompt, not just how much a request costs.

What Prompt Caching Actually Does

When a new request's beginning matches the beginning of a recent previous request closely enough, the platform can reuse the internal computation already done for that shared portion instead of redoing it, which is both faster and considerably cheaper for the portion that was reused.

system_instructions = """You are a customer support assistant for Acme Corp.
Always respond in a professional, empathetic tone. Never make promises about
refund amounts without checking the order database first. Format all monetary
values with two decimal places and a dollar sign...""" * 20  # a long, static block

def handle_support_message(client, customer_message: str):
    return client.responses.create(
        model="gpt-5.6-terra",
        input=[
            {"role": "system", "content": system_instructions},
            {"role": "user", "content": customer_message},
        ],
    )

Note: Whether prompt caching is automatic, what minimum prompt length triggers it, how long a cached prefix remains eligible for reuse, and how cache hits are reported can all vary by SDK version and platform configuration. Confirm the current caching behavior against the current official documentation before relying on a specific threshold or savings figure.

Here, system_instructions is identical on every call to handle_support_message(), while customer_message is the only part that actually changes — this is exactly the shape of prompt that benefits from caching, since the large static portion can be reused across every customer interaction instead of being reprocessed each time.

Checking Whether a Request Actually Hit the Cache

A response typically reports how many of its input tokens were served from cache, which is the only reliable way to confirm caching is actually happening rather than assuming it based on prompt structure alone.

response = client.responses.create(
    model="gpt-5.6-terra",
    input=[
        {"role": "system", "content": system_instructions},
        {"role": "user", "content": "Where is my order?"},
    ],
)

cached_tokens = getattr(response.usage, "cached_tokens", 0)
total_input_tokens = response.usage.input_tokens
print(f"{cached_tokens} of {total_input_tokens} input tokens were served from cache")

Note: The exact field name reporting cached token counts (cached_tokens here, or an equivalent under a different name) and where it appears on the usage object can vary by SDK version. Confirm the current field name against your installed SDK version's documentation.

Measuring this directly matters because caching eligibility depends on details — exact prefix match, recency, minimum length — that aren't always obvious from reading the prompt alone; an application that assumes it's benefiting from caching without ever checking cached_tokens might be paying full price for every request without realizing it.

Structuring a Prompt to Maximize Cache Reuse

Caching works by matching an identical prefix — the shared beginning of a request — so the order in which content appears in a prompt directly determines whether caching can help at all.

# Cache-friendly: static content first, dynamic content last.
def build_cache_friendly_prompt(customer_message: str) -> list:
    return [
        {"role": "system", "content": system_instructions},  # identical every time
        {"role": "user", "content": customer_message},         # changes every time
    ]

# Cache-defeating: dynamic content inserted before the static block.
def build_cache_defeating_prompt(customer_message: str, timestamp: str) -> list:
    return [
        {"role": "system", "content": f"Current time: {timestamp}\n\n{system_instructions}"},
        {"role": "user", "content": customer_message},
    ]

build_cache_defeating_prompt() looks almost identical to the cache-friendly version, but inserting a changing timestamp at the very beginning of the system message means the prefix is different on every single call — even though 99% of the system message's content is unchanged, the part that matches for caching purposes is the literal beginning of the string, and that beginning now differs every time. This is a subtle but consequential mistake: any content that changes between calls needs to go after the static portion, never before or inside it, for caching to have any effect.

Prompt Caching Is One Lever Among Several

Caching reduces the cost of reprocessing an unchanged prefix, but it doesn't address every source of cost in an application, and treating it as the only lever worth pulling misses several others that are often more impactful.

The most effective cost strategy for most applications combines several of these rather than relying on any single one — a support system, for instance, might use prompt caching for its shared instructions, a smaller model for an initial routing decision (mirroring Unit 11's triage agent), and the Batch API for a nightly summarization job that doesn't need an immediate response.

Trimming Context That Doesn't Earn Its Cost

Beyond caching what's already necessary, it's worth periodically checking whether everything currently being sent is actually needed — Unit 4's conversation state and Unit 9's retrieved documents both accumulate naturally over time, and neither automatically shrinks back down once it's no longer relevant.

def trim_conversation_history(messages: list, max_messages: int = 10) -> list:
    """Keep a system message (if present) plus only the most recent turns,
    rather than sending an ever-growing, mostly-irrelevant history."""
    system_messages = [m for m in messages if m["role"] == "system"]
    other_messages = [m for m in messages if m["role"] != "system"]
    return system_messages + other_messages[-max_messages:]

A long-running conversation that keeps every prior turn (as Unit 4's naive conversation-history examples did, before that unit introduced previous_response_id as an alternative) pays to reprocess an ever-larger amount of increasingly irrelevant context on every single turn — trimming to a bounded recent window, or relying on the platform's own conversation-state mechanism instead of resending full history yourself, keeps input size (and therefore cost) from growing unboundedly over a long session.

Common Mistakes

Placing dynamic content before or inside a prompt's static portion, defeating prompt caching entirely even though the majority of the prompt's content doesn't actually change between calls.

Assuming caching is happening without checking cached_tokens on the response, potentially paying full price for every request while believing costs are already optimized.

Treating prompt caching as the only available cost lever, missing larger savings available from model selection, reasoning effort, output length limits, or the Batch API for suitable workloads.

Sending an ever-growing, untrimmed conversation history on every turn, paying to reprocess increasingly irrelevant context as a session lengthens.

Best Practices

Structure prompts with static, shared content first and call-specific content last, preserving an identical prefix across calls so caching can actually take effect.

Measure actual cache hit rates from the response's usage data, rather than assuming a particular prompt structure is being cached without confirming it.

Combine multiple cost optimizations rather than relying on caching alone — model choice, reasoning effort, output limits, and batching each address a different source of cost.

Periodically trim conversation history and retrieved context to what the current turn actually needs, rather than letting input size grow unboundedly over a long-running session.