Production performance and cost review checklist

This lesson consolidates the observability, cost, and latency techniques from this unit into a review process: a structured checklist for periodically auditing a production application's performance and cost health. Unit 14's deployment checklist covers what must be true before shipping a feature — auth, error handling, rate limiting, rollback plans. This checklist is different in purpose and timing: it is run periodically on an already-running system to catch drift, waste, and emerging problems that a one-time deployment check cannot.

Why a Recurring Review Is Necessary

A system that was well-optimized at launch does not stay that way automatically. Usage patterns shift as the user base grows or changes. New features get added without their cost being checked against the patterns established for existing ones. Provider pricing and model lineups change over time (Lesson 6). A prompt accumulates small additions over months of iteration until it is significantly larger than it needs to be (Lesson 5). None of these are one-time problems that a launch checklist can catch — they are slow, cumulative drift that only a recurring review will surface.

The right cadence depends on the application's scale and rate of change: a fast-growing product with frequent feature releases benefits from a monthly review, while a stable, mature system might review quarterly. What matters more than the exact cadence is that the review actually happens on a schedule, rather than only in reaction to a surprising bill.

Building the Review as Executable Checks

A checklist that lives only as a document tends to be skipped under time pressure. Encoding each check as a function that inspects real data makes the review partially automatable and repeatable, and turns "did we check this" into an objective, testable question.

from dataclasses import dataclass
from enum import Enum


class CheckStatus(Enum):
    PASS = "pass"
    WARN = "warn"
    FAIL = "fail"


@dataclass
class ReviewResult:
    check_name: str
    status: CheckStatus
    detail: str


def check_cost_trend(cost_trend: list[float], max_growth_pct: float = 20.0) -> ReviewResult:
    if len(cost_trend) < 2 or cost_trend[0] == 0:
        return ReviewResult("cost_trend", CheckStatus.WARN, "Insufficient data to evaluate trend")

    growth_pct = ((cost_trend[-1] - cost_trend[0]) / cost_trend[0]) * 100
    if growth_pct > max_growth_pct:
        return ReviewResult(
            "cost_trend",
            CheckStatus.FAIL,
            f"Cost grew {growth_pct:.1f}% over the period, exceeding {max_growth_pct}% threshold",
        )
    return ReviewResult("cost_trend", CheckStatus.PASS, f"Cost grew {growth_pct:.1f}%, within threshold")


def check_error_rate(error_rate: float, max_error_rate: float = 0.02) -> ReviewResult:
    if error_rate > max_error_rate:
        return ReviewResult(
            "error_rate",
            CheckStatus.FAIL,
            f"Error rate {error_rate:.2%} exceeds {max_error_rate:.2%} threshold",
        )
    return ReviewResult("error_rate", CheckStatus.PASS, f"Error rate {error_rate:.2%} within threshold")

Each check function returns a structured ReviewResult rather than just printing a message or raising an exception, which lets the results be collected, filtered, and reported on programmatically — for example, showing only FAIL results in a summary, or tracking how many checks pass over successive reviews as a trend of its own. check_cost_trend's handling of insufficient data (len(cost_trend) < 2 or cost_trend[0] == 0) returning WARN rather than PASS or FAIL is a deliberate third outcome: it would be misleading to claim a trend "passed" a threshold check when there was not enough data to compute a trend at all.

Checking Cost-Per-Value Efficiency

Beyond raw cost growth, the review should check whether cost-per-unit-of-value (introduced in Lesson 3) is holding steady or degrading — a rising total cost that is matched by proportional growth in usage may be entirely healthy, while a rising cost-per-unit signals an efficiency regression.

def check_cost_efficiency(
    current_cost_per_unit: dict[str, float],
    baseline_cost_per_unit: dict[str, float],
    max_regression_pct: float = 15.0,
) -> list[ReviewResult]:
    results = []
    for feature, current in current_cost_per_unit.items():
        baseline = baseline_cost_per_unit.get(feature)
        if baseline is None or baseline == 0:
            results.append(ReviewResult(f"efficiency:{feature}", CheckStatus.WARN, "No baseline available"))
            continue

        change_pct = ((current - baseline) / baseline) * 100
        if change_pct > max_regression_pct:
            results.append(ReviewResult(
                f"efficiency:{feature}",
                CheckStatus.FAIL,
                f"Cost per unit rose {change_pct:.1f}% versus baseline",
            ))
        else:
            results.append(ReviewResult(
                f"efficiency:{feature}",
                CheckStatus.PASS,
                f"Cost per unit changed {change_pct:.1f}%, within tolerance",
            ))
    return results

This function compares each feature's current cost-per-unit against a stored baseline_cost_per_unit from a previous review, rather than against an arbitrary fixed number, because the acceptable cost-per-unit varies enormously by feature — there is no single meaningful global threshold. Using the feature's own prior value as its baseline means the check is really asking "did this get worse," which is the actually useful question, rather than "is this above some number picked without context."

Checking Model and Caching Configuration Drift

Some checks are not about metrics trending badly but about configuration that has silently drifted from what was intended — a caching layer that was added but never enabled in a particular environment, or a feature still pointed at an old model despite a newer, better-suited option being available.

def check_caching_enabled(feature_configs: dict[str, dict]) -> list[ReviewResult]:
    results = []
    for feature, config in feature_configs.items():
        if config.get("expected_cache_type") and not config.get("cache_enabled"):
            results.append(ReviewResult(
                f"caching:{feature}",
                CheckStatus.FAIL,
                f"Feature expects {config['expected_cache_type']} caching but it is not enabled",
            ))
        else:
            results.append(ReviewResult(f"caching:{feature}", CheckStatus.PASS, "Caching configuration as expected"))
    return results


def check_model_currency(feature_configs: dict[str, dict], deprecated_models: set[str]) -> list[ReviewResult]:
    results = []
    for feature, config in feature_configs.items():
        model = config.get("model")
        if model in deprecated_models:
            results.append(ReviewResult(
                f"model:{feature}",
                CheckStatus.FAIL,
                f"Feature uses deprecated model '{model}'",
            ))
        else:
            results.append(ReviewResult(f"model:{feature}", CheckStatus.PASS, f"Model '{model}' is current"))
    return results

check_caching_enabled treats a feature that should have caching (per its own configured expected_cache_type) but does not have it turned on as a failure — this catches a real and common drift scenario where caching is built and works in one environment but a configuration flag was never flipped on in another, or was accidentally disabled during a later change. check_model_currency checks against a maintained deprecated_models set rather than hardcoding a specific "current" model name, since the set of deprecated models is the more stable thing to maintain — new models are added to the ecosystem far more often than old ones are formally deprecated.

Assembling and Running the Full Review

The individual checks combine into a single review run that produces one consolidated report, which is what actually gets read at the end of a review cycle.

def run_full_review(
    cost_trend: list[float],
    error_rate: float,
    current_cost_per_unit: dict[str, float],
    baseline_cost_per_unit: dict[str, float],
    feature_configs: dict[str, dict],
    deprecated_models: set[str],
) -> list[ReviewResult]:
    results = [
        check_cost_trend(cost_trend),
        check_error_rate(error_rate),
    ]
    results.extend(check_cost_efficiency(current_cost_per_unit, baseline_cost_per_unit))
    results.extend(check_caching_enabled(feature_configs))
    results.extend(check_model_currency(feature_configs, deprecated_models))
    return results


def summarize_review(results: list[ReviewResult]) -> str:
    failures = [r for r in results if r.status == CheckStatus.FAIL]
    warnings = [r for r in results if r.status == CheckStatus.WARN]
    lines = [f"Review: {len(results)} checks, {len(failures)} failed, {len(warnings)} warnings"]
    for r in failures:
        lines.append(f"  FAIL: {r.check_name} - {r.detail}")
    for r in warnings:
        lines.append(f"  WARN: {r.check_name} - {r.detail}")
    return "\n".join(lines)

run_full_review is intentionally a thin composition of the independent check functions rather than a monolithic function with all the logic inline — this keeps each check independently testable (as shown next) and makes adding a new category of check, later, a matter of writing one new function and adding one line here, without touching the existing checks. summarize_review surfaces failures before warnings in the printed output, since failures represent an actual threshold breach requiring attention, while warnings typically just indicate missing data or context that a reviewer should be aware of but which may not require immediate action.

Testing the Review Checks

Each check function is pure and deterministic, and should be tested against both a passing and a failing scenario to confirm the threshold logic is correct in both directions.

def test_check_cost_trend_flags_excessive_growth():
    passing = check_cost_trend([100.0, 110.0], max_growth_pct=20.0)
    failing = check_cost_trend([100.0, 150.0], max_growth_pct=20.0)

    assert passing.status == CheckStatus.PASS
    assert failing.status == CheckStatus.FAIL
    print("PASS: cost trend check distinguishes acceptable from excessive growth")


def test_check_model_currency_flags_deprecated_models():
    configs = {
        "summarize": {"model": "gpt-5.6-terra"},
        "legacy_feature": {"model": "gpt-4-legacy"},
    }
    results = check_model_currency(configs, deprecated_models={"gpt-4-legacy"})

    by_name = {r.check_name: r for r in results}
    assert by_name["model:summarize"].status == CheckStatus.PASS
    assert by_name["model:legacy_feature"].status == CheckStatus.FAIL
    print("PASS: model currency check flags only the deprecated model")


test_check_cost_trend_flags_excessive_growth()
test_check_model_currency_flags_deprecated_models()

The first test uses two cost trends deliberately chosen relative to the same 20% threshold — a 10% increase and a 50% increase — so the test exercises both the pass and fail branches of the same function with the same threshold, confirming the boundary logic rather than just one arbitrary case. The second test mixes one current and one deprecated model in the same configs dictionary specifically to confirm the check correctly distinguishes between them rather than, for instance, failing everything whenever any deprecated model is present anywhere in the configuration.

Running the Review as an Operational Habit

The checklist this lesson builds is only valuable if it is actually run on a schedule and its output is actually acted upon. In practice, this means scheduling run_full_review to execute automatically (for example, as a weekly or monthly job) against real data pulled from the usage aggregation and dashboard infrastructure built in Lessons 3 and 9, with its summarize_review output delivered to whoever owns cost and performance for the application — rather than treating this as a one-time exercise performed once after reading this lesson.

Common Mistakes

Treating a performance and cost review as a one-time exercise. Usage patterns, model lineups, and pricing all change continuously; a review done once at launch has no ability to catch the drift that accumulates afterward.

Comparing cost against a fixed dollar figure instead of a relative baseline. A feature's acceptable cost varies enormously by what it does; comparing against its own historical baseline (as check_cost_efficiency does) is almost always more meaningful than an arbitrary absolute threshold.

Running checks manually from memory instead of encoding them as repeatable functions. A checklist that exists only as a mental habit or a static document gets skipped under deadline pressure; a checklist encoded as functions that read real data can be run consistently and even automated.

Best Practices

Automate what can be automated, and schedule the rest. Checks like cost trend and error rate can run entirely from logged data with no human judgment required; schedule these to run automatically and reserve human review time for interpreting the results and deciding on action.

Store review results historically, not just as a point-in-time report. Keeping past ReviewResult sets lets you see whether a given check has been failing repeatedly (a persistent, unaddressed problem) or failed once as an anomaly, which changes how urgently it should be treated.

Close the loop between review findings and the techniques from earlier lessons. A failed cost-efficiency check should lead directly to applying prompt trimming (Lesson 5), model reselection (Lesson 6), or caching (Lesson 7) — a review that identifies problems but never triggers a fix provides no real value beyond the observation itself.