Caching strategies for repeated AI work

Lesson 4 introduced application-level response caching as one technique for reducing unnecessary model calls. This lesson goes deeper into caching as a discipline of its own: the different layers at which caching can happen, how to choose cache keys and expiry policies correctly, and how to keep a cache from serving stale or wrong results.

The Layers of Caching Available

There are three distinct places caching can happen in an LLM-backed application, and they are not interchangeable:

• Provider-side prompt caching (Unit 12, Lesson 4) — the provider caches the internal representation of a repeated prompt prefix, discounting the cost of reprocessing it. This still executes a model call; it only makes the input-processing portion of that call cheaper and faster.
• Application-level response caching (introduced in Lesson 4 of this unit) — your application stores the final output of a model call, keyed by a normalized version of the input, and can return it without calling the model at all on a hit.
• Semantic caching — a more advanced variant of response caching that matches on the meaning of a request rather than its exact text, so that a paraphrased question ("What's your refund policy?" versus "How do refunds work?") can still hit a cached answer.

Each layer catches a different kind of repetition. Provider-side caching is essentially free to enable and helps with any repeated prefix, whether or not the surrounding text varies. Application-level caching requires you to build and maintain the cache, but it can eliminate the model call entirely for exact repeats. Semantic caching catches the widest range of repetition but is the most complex to build correctly and carries the highest risk of serving a wrong answer, since "similar enough" is a judgment call.

Choosing the Right Cache Key

The cache key determines what counts as "the same request." Getting this wrong is the most common source of caching bugs — either a cache that never hits because keys are too specific, or one that returns wrong answers because keys are too permissive.

import hashlib
import json


def build_cache_key(feature: str, params: dict, user_scoped: bool = False, user_id: str | None = None) -> str:
    key_data = {"feature": feature, "params": params}
    if user_scoped:
        if user_id is None:
            raise ValueError("user_id is required when user_scoped=True")
        key_data["user_id"] = user_id

    canonical = json.dumps(key_data, sort_keys=True)
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

The user_scoped flag is the key design decision here: some cached results are safe to share across all users (a translation of a fixed UI string, a summary of a public document), while others must never be shared across users even if the input text is identical (a personalized recommendation, anything that could leak one user's data into another user's response). Raising a ValueError when user_scoped=True but no user_id is provided fails loudly at the call site rather than silently producing a key that accidentally omits the user scope — a bug here would be a serious cross-user data leak, so this function is intentionally strict about it rather than defaulting to a permissive behavior.

Setting Expiry Based on Content Volatility

Not all cached content should live for the same amount of time. The correct expiry depends on how quickly the correct answer changes, not on a single global default applied to everything.

from dataclasses import dataclass
from enum import Enum


class Volatility(Enum):
    STABLE = "stable"        # unlikely to change: static translations, fixed reference text
    DAILY = "daily"          # changes at most once a day: summaries of daily-updated content
    VOLATILE = "volatile"    # changes frequently: anything tied to live or personalized data


EXPIRY_SECONDS = {
    Volatility.STABLE: 60 * 60 * 24 * 30,   # 30 days
    Volatility.DAILY: 60 * 60 * 12,         # 12 hours
    Volatility.VOLATILE: 60 * 5,            # 5 minutes
}


@dataclass
class CacheEntry:
    value: str
    expires_at: float


def is_expired(entry: CacheEntry, now: float) -> bool:
    return now >= entry.expires_at

Classifying content by Volatility rather than picking one expiry for the whole cache reflects a real trade-off: a longer expiry increases the hit rate (more requests are served from cache) but increases the risk of serving an answer that is no longer correct, while a shorter expiry reduces that risk but also reduces the hit rate and therefore the cost savings. Assigning VOLATILE a five-minute expiry rather than caching it at all is a middle ground — it still catches near-simultaneous duplicate requests (similar to the deduplication technique in Lesson 4) without risking a long-lived stale answer for content that changes often.

Building a Cache With Expiry and Invalidation

Combining the cache key and expiry concepts into a working cache requires handling three operations correctly: reading a possibly-expired entry, writing a new entry with the right expiry, and explicitly invalidating an entry when the underlying data changes.

import time


class TieredCache:
    def __init__(self):
        self._store: dict[str, CacheEntry] = {}

    def get(self, key: str) -> str | None:
        entry = self._store.get(key)
        if entry is None:
            return None
        if is_expired(entry, time.time()):
            del self._store[key]
            return None
        return entry.value

    def set(self, key: str, value: str, volatility: Volatility) -> None:
        ttl = EXPIRY_SECONDS[volatility]
        self._store[key] = CacheEntry(value=value, expires_at=time.time() + ttl)

    def invalidate(self, key: str) -> None:
        self._store.pop(key, None)

get deletes an expired entry as soon as it is discovered rather than leaving it in place, which keeps the cache from accumulating stale entries indefinitely between reads — this is a common and simple approach called lazy expiry, appropriate for caches that are read often enough that stale entries get cleaned up naturally without needing a separate background sweep. invalidate exists as an explicit operation because expiry alone is not always sufficient: if the underlying source data changes (a document is edited, a price updates) before the cache entry's natural expiry, the application needs a way to remove that specific entry immediately rather than waiting out the TTL and serving a wrong answer in the meantime.

Semantic Caching: Matching on Meaning, Not Exact Text

Exact-match caching, as built above, misses a large class of genuine duplicates: two users asking the same underlying question in different words. Semantic caching addresses this by comparing embeddings — numeric representations of meaning — rather than exact strings.

def cosine_similarity(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = sum(x * x for x in a) ** 0.5
    norm_b = sum(y * y for y in b) ** 0.5
    if norm_a == 0 or norm_b == 0:
        return 0.0
    return dot / (norm_a * norm_b)


class SemanticCache:
    def __init__(self, similarity_threshold: float = 0.92):
        self._entries: list[tuple[list[float], str]] = []  # (embedding, response)
        self._similarity_threshold = similarity_threshold

    def find_match(self, query_embedding: list[float]) -> str | None:
        best_score = 0.0
        best_response = None
        for embedding, response in self._entries:
            score = cosine_similarity(query_embedding, embedding)
            if score > best_score:
                best_score = score
                best_response = response
        if best_score >= self._similarity_threshold:
            return best_response
        return None

    def add(self, embedding: list[float], response: str) -> None:
        self._entries.append((embedding, response))

similarity_threshold is the single most important tuning parameter for a semantic cache, and it is a genuine precision-versus-recall trade-off: a lower threshold catches more paraphrased duplicates (higher hit rate) but risks matching two questions that are similar in wording but different in actual intent, returning a wrong cached answer with no indication anything went wrong. A higher threshold is safer but catches fewer real duplicates. Because a wrong semantic-cache hit is a silent correctness bug — the user gets a confidently wrong answer with no error raised — semantic caching should generally start with a conservative (high) threshold and only be loosened after measuring false-positive matches on real traffic, never the other way around.

Testing Cache Behavior

Cache correctness — expiry, invalidation, and key uniqueness — is exactly the kind of logic that benefits from deterministic tests using a controllable clock, rather than relying on real elapsed time.

def test_cache_expires_entries_after_ttl():
    cache = TieredCache()
    fake_now = {"t": 1000.0}

    def fake_time():
        return fake_now["t"]

    global time
    original_time_time = time.time
    time.time = fake_time

    try:
        cache.set("key1", "value1", Volatility.VOLATILE)  # 5 minute TTL
        assert cache.get("key1") == "value1"

        fake_now["t"] += 60 * 6  # advance 6 minutes, past the 5 minute TTL
        assert cache.get("key1") is None
        print("PASS: cache entry expires after its TTL elapses")
    finally:
        time.time = original_time_time


def test_semantic_cache_matches_above_threshold_only():
    cache = SemanticCache(similarity_threshold=0.9)
    cache.add([1.0, 0.0], "cached answer")

    close_match = cache.find_match([0.95, 0.05])
    far_match = cache.find_match([0.0, 1.0])

    assert close_match == "cached answer"
    assert far_match is None
    print("PASS: semantic cache matches similar vectors and rejects dissimilar ones")


test_cache_expires_entries_after_ttl()
test_semantic_cache_matches_above_threshold_only()

The first test replaces time.time with a controllable fake so the TTL expiry can be tested deterministically, without an actual six-minute time.sleep — a common pattern for testing any time-dependent logic quickly and reliably. The second test picks vectors specifically chosen to be either clearly similar ([0.95, 0.05] versus [1.0, 0.0]) or clearly dissimilar ([0.0, 1.0] versus [1.0, 0.0]), so the threshold behavior is unambiguous rather than relying on borderline values that could make the test flaky.

Common Mistakes

Caching personalized or user-specific results without scoping the cache key by user. This is a data leakage bug, not just a correctness bug — one user's personalized response can be served to a different user, which is a serious privacy failure.

Setting one global expiry for all cached content regardless of how quickly it changes. A single TTL is either too long for volatile content (serving stale answers) or too short for stable content (giving up hit rate and cost savings for no reason).

Deploying semantic caching with a similarity threshold that has not been validated against real near-miss examples. An untested threshold can silently serve wrong answers for questions that are superficially similar but have different correct answers, with no visible error to indicate the mistake.

Best Practices

Match the caching layer to the kind of repetition you actually have. Use provider-side prompt caching for anything with a stable, repeated prefix; use application-level response caching for exact repeated requests; reserve semantic caching for cases where paraphrased repeats are common enough to be worth its added complexity and risk.

Always make cache invalidation possible, not just expiry. Relying solely on TTL-based expiry means a stale entry can persist for its full TTL even after the underlying data has changed; an explicit invalidate path lets you correct that immediately when you know a change occurred.

Monitor cache hit rate and false-positive rate as ongoing metrics. A cache that never measures its own hit rate cannot be tuned, and a semantic cache that never measures false-positive matches (wrong answers served due to over-loose similarity) can silently degrade answer quality without anyone noticing.