Reduce Repeated Input Costs

Caching Is Not Automatically Cheaper

Lessons 121–125 showed that cache reads cost a fraction of normal input. That is only half of the economics. A cache write costs more than normal input, and a cache entry that expires before it is read has been paid for and never used. Caching saves money only when the same prefix is read enough times to pay back its write.

This lesson calculates that break-even point, shows how request timing decides whether caching helps, and lists the design choices that reduce repeated input cost in general.

The Break-Even Point

Let the base input price be 1. A prefix written once costs the write multiple w, and each later read costs the read multiple r. Without caching, each request costs 1 for that prefix.

After the write, with n reads:

cost with caching    = w + n × r
cost without caching = 1 + n × 1

Caching is cheaper when w + n × r < 1 + n, which rearranges to:

n > (w − 1) / (1 − r)

For the current multipliers, this gives:

The 5-minute cache pays back after a single reuse. The 1-hour cache needs two reuses, because its write costs twice the base price.

Timing Decides the Outcome

The break-even calculation assumes the cache stays alive between reads. Its lifetime is 5 minutes by default, or 1 hour with the extended TTL, and each read refreshes it (Lesson 130 covers the details). That creates three situations:

The second row is the costly mistake. A 5-minute cache on a feature used every 10 minutes writes on every request, paying the 1.25× premium each time for nothing.

The example below makes this concrete.

Practical Example: Comparing Caching Strategies

This script estimates the input cost of 100 requests that share a 50,000-token prefix, under four scenarios. Each scenario is simulated request by request, so the write and read logic follows the TTL rules.

"""cache_economics.py: when caching saves money, and when it costs more."""

import math

BASE_PRICE = 2.00            # USD per million tokens (Sonnet 5.5)
REQUESTS = 100
PREFIX_TOKENS = 50_000


def breakeven_reads(write_multiple: float, read_multiple: float) -> int:
    """Smallest number of reads for which caching costs less than not caching."""
    return math.floor((write_multiple - 1) / (1 - read_multiple)) + 1


def scenario_cost(gap_minutes: float, ttl_minutes: float, write_multiple: float,
                  read_multiple: float) -> float:
    """Total prefix cost in USD, simulating one request after another."""
    total_tokens_cost = 0.0
    cache_alive = False
    for _ in range(REQUESTS):
        if not cache_alive or gap_minutes > ttl_minutes:
            total_tokens_cost += PREFIX_TOKENS * BASE_PRICE * write_multiple
            cache_alive = True
        else:
            total_tokens_cost += PREFIX_TOKENS * BASE_PRICE * read_multiple
    return total_tokens_cost / 1_000_000


if __name__ == "__main__":
    print(f"Break-even reads, 5-minute TTL: {breakeven_reads(1.25, 0.10)}")
    print(f"Break-even reads, 1-hour TTL: {breakeven_reads(2.00, 0.10)}")

    no_cache = REQUESTS * PREFIX_TOKENS * BASE_PRICE / 1_000_000
    print(f"no caching: ${no_cache:.3f}")
    print(f"5-minute TTL, a request every minute: "
          f"${scenario_cost(1, 5, 1.25, 0.10):.3f}")
    print(f"5-minute TTL, a request every 10 minutes: "
          f"${scenario_cost(10, 5, 1.25, 0.10):.3f}")
    print(f"1-hour TTL, a request every 10 minutes: "
          f"${scenario_cost(10, 60, 2.00, 0.10):.3f}")

Output:

Break-even reads, 5-minute TTL: 1
Break-even reads, 1-hour TTL: 2
no caching: $10.000
5-minute TTL, a request every minute: $0.620
5-minute TTL, a request every 10 minutes: $12.500
1-hour TTL, a request every 10 minutes: $0.695

How It Works

1. breakeven_reads() solves the inequality from the previous section. The floor(...) + 1 gives the smallest whole number of reads that is strictly greater than the threshold.
2. scenario_cost() simulates each request. A request writes the prefix if the cache is not alive, or if the gap since the last request is longer than the TTL. Otherwise it reads the prefix.
3. The 5-minute cache with requests every minute writes once and reads 99 times: $0.125 for the write, plus $0.495 for the reads, which is $0.62. Caching saves about 94% of the $10.00 baseline.
4. The 5-minute cache with requests every 10 minutes writes on every request, because each gap exceeds the TTL. That is 100 writes at $0.125 each, $12.50 in total: more than the $10.00 without caching.
5. The 1-hour cache with requests every 10 minutes writes once at the 2× price, then reads 99 times. It costs $0.695, close to the frequent-request case, and far below the mistake in step 4.

Note: the simulation ignores the small uncached question in each request, which is the same in every scenario. It also assumes steady prices and gaps. Real traffic is irregular, so measure the hit rate in production (Lesson 125).

Other Ways to Reduce Repeated Input

Caching is only one lever. Several design choices reduce the input that must be sent at all:

Caching makes repeated input cheap, but output tokens are not reduced by caching. Output is typically priced several times higher than input, so keeping answers concise (Lessons 38 and 47) often matters as much as caching.

Common Mistakes

Caching Everything

What happens: writes are paid for on content that is never read again.

Fix: cache prefixes that are reused, and leave one-off content uncached.

Using the 5-Minute TTL for Infrequent Requests

What happens: every request writes, and costs more than no caching at all.

Fix: measure the gap between requests, and use the 1-hour TTL when gaps fall between 5 and 60 minutes.

Ignoring Break-Even Reads

What happens: a feature with one or two requests per prefix is cached, and the write premium is never recovered.

Fix: estimate reads per prefix before turning caching on.

Optimizing Input and Forgetting Output

What happens: input costs fall, but long answers dominate the bill.

Fix: budget output tokens separately (Lesson 47).

Best Practices

• Estimate reads per prefix before enabling caching.
• Match the TTL to request spacing: 5 minutes for frequent traffic, 1 hour for spaced traffic.
• Keep prefixes stable and dynamic content last.
• Send only the tools and history a request needs.
• Measure hit rates and compare them with the break-even estimate.
• Budget output tokens separately from input.

Exercise

1. Change the gap in the 5-minute scenario to 4 minutes. Does the cost change from the one-minute case? Why?
2. Change REQUESTS to 2, and compare caching with no caching. Which is cheaper, and by how much?
3. Compute the break-even number of reads for a write multiple of 1.25 and a read multiple of 0.025 (Fable 5.1). Is the answer different from Opus 5.5?
4. Explain why an application that sends one request per user per session might gain nothing from the 5-minute cache, and might lose money.

Step 1: no change. A gap of 4 minutes is shorter than the 5-minute TTL, so the cache stays alive, and each request reads the prefix after the first write. Only gaps longer than the TTL cause writes. This is why the TTL, not the average request rate, decides the outcome.

Step 2: with two requests, no caching costs $0.200 (2 × 50,000 tokens at $2.00 per million). Caching costs a write of $0.125 plus one read of $0.005, which is $0.130. Caching is cheaper by $0.070, about 35%. The saving is smaller than in the 100-request case, because the write premium is spread over fewer reads.