Reducing unnecessary model calls

The cheapest model call is the one that never happens. Before optimizing what a model call costs (Lessons 2 and 5) or which model handles it (Lesson 6), it is worth eliminating calls that provide no additional value at all. This lesson covers three concrete techniques: short-circuiting obvious cases, deduplicating identical or near-identical requests, and caching results outside the model so repeated work is never resent.

Why This Comes Before Prompt and Cost Optimization

Reducing token count or switching to a cheaper model both reduce the cost of a call that still happens. Eliminating the call entirely reduces cost to zero for that request, and also removes its latency, its contribution to rate limits, and its exposure to model failure modes. In a system under load, cutting the number of calls is often more impactful than cutting the cost of each one, because it also protects capacity: fewer calls means more headroom before you hit provider rate limits.

This does not mean every call is avoidable — a request that genuinely needs the model's reasoning or generation ability must go through. The goal is to identify the subset of calls that are redundant, predictable, or unnecessary, and remove exactly those.

Short-Circuiting Obvious Cases

Many applications route every request through the model, even when a simple rule could resolve a meaningful fraction of cases without one. A classification feature that routes customer messages, for example, might not need a model call at all for a message that is obviously spam, or one that exactly matches a known FAQ question.

import re

KNOWN_SPAM_PATTERNS = [
    re.compile(r"\bfree\s+crypto\b", re.IGNORECASE),
    re.compile(r"\bclick\s+here\s+now\b", re.IGNORECASE),
]

FAQ_EXACT_MATCHES = {
    "what are your business hours": "We are open Monday to Friday, 9am to 6pm.",
    "how do i reset my password": "Go to Settings > Security > Reset Password.",
}


def classify_message(message: str) -> str | None:
    """Return a short-circuit classification, or None if a model call is needed."""
    normalized = message.strip().lower()

    if normalized in FAQ_EXACT_MATCHES:
        return FAQ_EXACT_MATCHES[normalized]

    for pattern in KNOWN_SPAM_PATTERNS:
        if pattern.search(message):
            return "[spam - filtered]"

    return None


def handle_message(message: str) -> str:
    short_circuit_result = classify_message(message)
    if short_circuit_result is not None:
        return short_circuit_result
    return call_model_for_response(message)

classify_message returns None specifically to signal "no rule applied, fall through to the model" — using None as a sentinel rather than an empty string keeps the two outcomes (a real short-circuit answer versus "no match") unambiguous, which matters because a legitimate response could itself be an empty string in some edge cases. This pattern only pays off when the rules are cheap to evaluate (regex matching and dictionary lookups run in microseconds) and reliably correct — a rule that misclassifies real user intent creates a worse experience than simply calling the model, so short-circuit rules should be conservative and only fire on genuinely unambiguous cases, with everything else falling through.

Deduplicating Identical Requests

In many applications, especially those handling bursts of traffic (batch imports, retries, or multiple users asking similar questions), the exact same request can arrive more than once within a short window. Deduplication catches these before they reach the model.

import hashlib
import time


class RequestDeduplicator:
    def __init__(self, window_seconds: float = 5.0):
        self._window_seconds = window_seconds
        self._recent: dict[str, tuple[float, str]] = {}

    def _key(self, feature: str, prompt: str) -> str:
        raw = f"{feature}:{prompt}"
        return hashlib.sha256(raw.encode("utf-8")).hexdigest()

    def get_cached_result(self, feature: str, prompt: str) -> str | None:
        key = self._key(feature, prompt)
        entry = self._recent.get(key)
        if entry is None:
            return None
        timestamp, result = entry
        if time.time() - timestamp > self._window_seconds:
            del self._recent[key]
            return None
        return result

    def store_result(self, feature: str, prompt: str, result: str) -> None:
        key = self._key(feature, prompt)
        self._recent[key] = (time.time(), result)

Hashing the combination of feature and prompt with sha256 produces a fixed-size key regardless of prompt length, which keeps the lookup dictionary's memory usage predictable even if individual prompts are long. The window_seconds expiry is essential: without it, _recent would keep every distinct request forever, defeating deduplication's purpose of catching near-simultaneous duplicates (a double-submitted form, a retried request after a timeout) rather than being used as a general-purpose long-term cache, which is a distinct technique covered next.

Note the deliberate scope limitation here: this deduplicator is appropriate for catching accidental, near-term repeats — not for caching results the application expects to reuse over hours or days. Conflating short-term deduplication with long-term caching leads to either serving stale results (window too long) or missing genuine duplicates (window too short).

Caching Results Outside the Model

A distinct and often higher-impact technique is caching the result of a model call at the application layer, keyed by a normalized version of the input, so that a repeated request — potentially from a different user, hours or days later — never reaches the model at all. This is different from the prompt-prefix caching covered in Unit 12, Lesson 4, which speeds up and discounts the input processing of a request that still executes; application-level result caching skips the model call entirely when the answer is already known.

import json
from typing import Callable


class ResponseCache:
    def __init__(self):
        self._store: dict[str, str] = {}

    def _normalize(self, feature: str, params: dict) -> str:
        # Sort keys so equivalent params in a different order produce the same key.
        canonical = json.dumps({"feature": feature, "params": params}, sort_keys=True)
        return canonical

    def get_or_compute(
        self,
        feature: str,
        params: dict,
        compute: Callable[[], str],
    ) -> str:
        key = self._normalize(feature, params)
        if key in self._store:
            return self._store[key]
        result = compute()
        self._store[key] = result
        return result

Sorting dictionary keys with sort_keys=True before hashing or storing is a small detail with an outsized effect: without it, {"lang": "en", "text": "hi"} and {"text": "hi", "lang": "en"} — which represent the identical logical request — would produce different cache keys and never hit each other, silently defeating the cache for a large fraction of otherwise-identical requests. The get_or_compute method takes compute as a callable rather than calling the model directly, which decouples the cache from any specific model client and makes it trivially testable with a fake compute function, as shown below.

This technique is most valuable for requests with a naturally small, repeating space of inputs: translating a fixed set of UI strings, summarizing a document that many users will view, or answering a frequently-asked question that is not an exact string match (unlike the FAQ short-circuit above, this can cache the model's answer to a paraphrased version of a common question, once it has been asked and normalized). It is a poor fit for genuinely unique, personalized requests — caching a customer's individual support ticket response will almost never produce a cache hit, since inputs are rarely repeated verbatim.

Testing Call-Reduction Logic

None of these techniques should be tested against a real model. Each is a pure function or a small stateful object that can be tested with fakes and fixed inputs.

def test_classify_message_short_circuits_faq():
    result = classify_message("What are your business hours")
    assert result == "We are open Monday to Friday, 9am to 6pm."
    print("PASS: FAQ short-circuit matches without a model call")


def test_classify_message_falls_through_for_unknown_input():
    result = classify_message("Can you help me plan a trip to Kyoto?")
    assert result is None
    print("PASS: unrecognized input falls through to the model")


def test_response_cache_avoids_recomputation():
    cache = ResponseCache()
    call_count = {"count": 0}

    def fake_compute() -> str:
        call_count["count"] += 1
        return "translated text"

    first = cache.get_or_compute("translate", {"text": "hi", "lang": "es"}, fake_compute)
    second = cache.get_or_compute("translate", {"lang": "es", "text": "hi"}, fake_compute)

    assert first == "translated text"
    assert second == "translated text"
    assert call_count["count"] == 1, "compute should only run once due to caching"
    print("PASS: response cache serves the second call without recomputing")


test_classify_message_short_circuits_faq()
test_classify_message_falls_through_for_unknown_input()
test_response_cache_avoids_recomputation()

test_response_cache_avoids_recomputation deliberately calls get_or_compute twice with the params dictionary keys in a different order ({"text": "hi", "lang": "es"} versus {"lang": "es", "text": "hi"}) specifically to verify the sort_keys=True normalization works — this is exactly the kind of subtle bug that a test with identically-ordered dictionaries would never catch. The call_count dictionary acts as a simple mutable counter captured by the closure fake_compute, letting the test assert that compute ran exactly once even though get_or_compute was called twice.

Common Mistakes

Deduplicating by exact string match only, missing semantically identical requests. Two prompts that differ only in whitespace, capitalization, or key order in a structured payload are logically the same request but will miss a naive string-equality cache. Normalizing input (trimming, lowercasing where appropriate, sorting structured keys) before keying the cache catches far more genuine duplicates.

Using a single cache for both short-term deduplication and long-term result caching. These have different correctness requirements — deduplication needs a short expiry to avoid serving stale near-duplicate answers, while result caching for stable content can be kept much longer. Conflating them leads to picking a compromise window that serves neither purpose well.

Adding short-circuit rules that are too aggressive. A rule that fires on ambiguous input to avoid a model call can produce a wrong answer with high confidence, which is worse for the user than the latency and cost of an actual model call. Keep short-circuit conditions narrow and easy to verify as correct.

Best Practices

Measure short-circuit and cache hit rates. Track what fraction of requests are resolved without a model call, per technique, using the usage metrics framework from Lesson 3 — this tells you which technique is worth investing further effort in and which is providing negligible benefit.

Make cache and deduplication keys deterministic and normalized. Canonicalize inputs (sorted keys, trimmed whitespace, consistent casing) before hashing, so that logically identical requests reliably produce the same key regardless of superficial formatting differences.

Keep short-circuit and cache logic close to the request entry point. Placing these checks early, before any prompt construction or context assembly happens, avoids doing unnecessary work (like an expensive database lookup to build a prompt) for a request that will not actually reach the model.