Choosing models based on quality, speed, and cost

A provider typically offers several models within a family — a large, high-quality flagship model and one or more smaller, faster, cheaper variants. Choosing which one to use for a given feature is a genuine engineering decision with measurable trade-offs, not a default you set once and forget.

The Quality, Speed, and Cost Triangle

The three properties that differentiate models are quality (how good and reliable the output is for a given task), speed (latency per request), and cost (price per token). In practice, these three properties trade off against each other: a larger model tends to produce higher-quality output but is slower and more expensive per token; a smaller model is faster and cheaper but may produce lower-quality or less reliable output on harder tasks.

This trade-off is not fixed across all tasks. A smaller model might perform identically to a larger one on a simple, well-defined task (classifying sentiment, extracting a date from text) while performing noticeably worse on a task requiring multi-step reasoning or nuanced judgment. The right choice depends entirely on what the specific feature needs, which means model selection should be made per feature, not once for the entire application.

Why "Always Use the Best Model" Is Usually Wrong

It is tempting to default every feature to the largest, highest-quality model available, reasoning that quality is what matters most. This reasoning breaks down for three reasons. First, cost scales with usage — a feature called a million times a day at the flagship model's price can cost orders of magnitude more than the same feature on a smaller model, and if the task does not need the extra quality, that cost buys nothing. Second, latency matters for user experience (covered in depth in Lesson 8) — a slower model directly degrades an interactive feature even if its answers are marginally better. Third, using the largest model everywhere makes it harder to notice when a smaller model would have been entirely adequate, since there is no comparison baseline being collected.

The corollary is also true: defaulting everything to the cheapest, smallest model to minimize cost is equally wrong when the task genuinely requires stronger reasoning, and the cost of a wrong or low-quality answer (a bad recommendation, an incorrect summary that misleads a user) exceeds the cost difference between models.

A Framework for Model Selection

A structured way to make this decision per feature is to score each candidate model along the three dimensions for the specific task, using real evaluation data rather than intuition.

from dataclasses import dataclass


@dataclass
class ModelCandidate:
    name: str
    quality_score: float      # 0-1, from evaluation against a labeled test set
    avg_latency_ms: float
    cost_per_request: float


def select_model(
    candidates: list[ModelCandidate],
    min_quality: float,
    max_latency_ms: float,
) -> ModelCandidate | None:
    """Among candidates meeting minimum quality and latency bars, pick the cheapest."""
    eligible = [
        c for c in candidates
        if c.quality_score >= min_quality and c.avg_latency_ms <= max_latency_ms
    ]
    if not eligible:
        return None
    return min(eligible, key=lambda c: c.cost_per_request)

This function encodes a specific, deliberate policy: quality and latency are treated as hard constraints (a candidate that fails either bar is excluded entirely), and cost is the tiebreaker among everything that clears both bars. This ordering matters — it reflects the idea that quality and responsiveness below a certain threshold make a feature unusable regardless of how cheap it is, while above those thresholds, the cheapest option is the right default since additional quality above the bar may not translate into additional user value. The function returning None when no candidate is eligible is a deliberate signal that the requirements as stated cannot currently be met by any available model — this should be surfaced to whoever set the thresholds, not silently ignored by falling back to some default.

Note: quality_score here assumes you have already built an evaluation set for the task — a labeled sample of representative inputs with expected or acceptable outputs, scored by a rubric or comparison against known-good answers. Without a real evaluation set, quality claims about a model are just opinion; building this evaluation set is a prerequisite for this framework to work, not something the framework provides for you.

Running an A/B Comparison Between Models

Rather than trusting a single evaluation run, a more robust approach is to route a fraction of live traffic to a candidate model and compare its measured quality, latency, and cost against the current default under real conditions.

import random


class ModelRouter:
    def __init__(self, primary: str, candidate: str, candidate_traffic_pct: float = 0.05):
        self.primary = primary
        self.candidate = candidate
        self.candidate_traffic_pct = candidate_traffic_pct

    def choose_model(self) -> str:
        if random.random() < self.candidate_traffic_pct:
            return self.candidate
        return self.primary

Routing only a small percentage (candidate_traffic_pct, defaulting to 5%) of traffic to the candidate model limits the blast radius if the candidate performs worse than expected on real traffic, while still gathering enough real-world data to make a confident decision. This is the same underlying idea as a canary deployment in general software engineering, applied to model selection: validate on a small slice of production traffic before committing to a full rollout. Combined with the request logging from Lesson 1 (which already records which model served each request), comparing the candidate's and the primary's logged latency, error rate, and cost after enough traffic has accumulated gives a real, unbiased comparison — free of the sampling bias that a curated offline evaluation set can introduce.

Task-Specific Routing

A more advanced pattern than picking one model per feature is picking a model per request, based on some cheap-to-compute signal about how difficult that particular request is likely to be.

def estimate_task_difficulty(user_message: str) -> str:
    """A cheap heuristic classifier — not a model call — to route by difficulty."""
    word_count = len(user_message.split())
    has_multiple_questions = user_message.count("?") > 1
    mentions_complex_keywords = any(
        kw in user_message.lower()
        for kw in ["compare", "analyze", "explain why", "trade-off"]
    )

    if word_count > 80 or has_multiple_questions or mentions_complex_keywords:
        return "complex"
    return "simple"


def route_by_difficulty(user_message: str) -> str:
    difficulty = estimate_task_difficulty(user_message)
    if difficulty == "complex":
        return "gpt-5.6-terra"
    return "gpt-5.6-terra-mini"

estimate_task_difficulty is deliberately a cheap heuristic — string length, punctuation counting, keyword matching — rather than a model call, because using a model to decide which model to use would add cost and latency to every request, defeating the purpose. This kind of heuristic routing is inherently imperfect: it will sometimes send a genuinely complex short question to the smaller model, or an easy long question to the larger one. It is worth deploying only when measurement (again, via the request logs from Lesson 1) confirms the aggregate cost savings from correctly-routed simple requests outweigh the quality cost of the occasional misroute. When the heuristic is unreliable, a better middle ground is to let the smaller model attempt the request first and escalate to the larger model only if it signals low confidence or fails to satisfy the request — an approach worth validating against your specific task before adopting.

Testing Model Selection Logic

Because select_model and estimate_task_difficulty are pure functions of their inputs, they can be tested exhaustively with constructed candidates and messages, without any real model call.

def test_select_model_prefers_cheapest_among_eligible():
    candidates = [
        ModelCandidate("large", quality_score=0.95, avg_latency_ms=1200, cost_per_request=0.02),
        ModelCandidate("medium", quality_score=0.90, avg_latency_ms=600, cost_per_request=0.008),
        ModelCandidate("small", quality_score=0.70, avg_latency_ms=200, cost_per_request=0.001),
    ]
    chosen = select_model(candidates, min_quality=0.85, max_latency_ms=1000)
    assert chosen is not None and chosen.name == "medium"
    print("PASS: cheapest eligible model is chosen when quality and latency bars are met")


def test_select_model_returns_none_when_no_candidate_qualifies():
    candidates = [
        ModelCandidate("small", quality_score=0.60, avg_latency_ms=200, cost_per_request=0.001),
    ]
    chosen = select_model(candidates, min_quality=0.85, max_latency_ms=1000)
    assert chosen is None
    print("PASS: no eligible candidate correctly returns None instead of a default")


def test_route_by_difficulty_flags_long_complex_message():
    complex_message = "Can you compare these two approaches and explain why one has a better trade-off? " * 3
    assert route_by_difficulty(complex_message) == "gpt-5.6-terra"

    simple_message = "What time zone is Tokyo in?"
    assert route_by_difficulty(simple_message) == "gpt-5.6-terra-mini"
    print("PASS: difficulty routing sends complex and simple messages to different models")


test_select_model_prefers_cheapest_among_eligible()
test_select_model_returns_none_when_no_candidate_qualifies()
test_route_by_difficulty_flags_long_complex_message()

test_select_model_returns_none_when_no_candidate_qualifies is worth calling out specifically: it verifies the function's explicit failure mode, not just its happy path. A model-selection function that silently degrades to some fallback when no candidate qualifies would hide a real problem — that the current requirements cannot be met by any available model — behind an unremarkable-looking function call, exactly the kind of failure that goes unnoticed until it causes a production quality issue.

Common Mistakes

Choosing a model once at project start and never revisiting it. Provider model lineups change frequently — new models are released, older ones deprecated, prices adjusted — and a choice that was correct a year ago may no longer be optimal or even the best available trade-off today.

Evaluating model quality only on a small number of hand-picked examples. A handful of examples chosen because they look impressive is not a representative evaluation set; conclusions drawn from it will not generalize to real traffic, which is far more varied.

Ignoring latency when comparing models for interactive features. A model with marginally higher quality scores but twice the latency can produce a net negative user experience for an interactive feature, even though an offline quality comparison alone would recommend it.

Best Practices

Build a real evaluation set per feature before comparing models. A representative, labeled sample of realistic inputs is what turns a model comparison from guesswork into a measurable decision, and it does not need to be large to be useful — even a few dozen well-chosen examples beats none.

Validate model changes on live traffic before a full rollout. A canary-style comparison, as shown with ModelRouter, catches real-world quality or reliability issues that an offline evaluation set may have missed.

Re-evaluate model choice whenever the provider updates their lineup or pricing. Treat model selection as an ongoing operational decision tied to the usage metrics from Lesson 3, not a one-time architectural choice made at project start.