Why Prompts and Models Need Their Own Kind of Regression Test

In conventional software, a regression test protects against a code change accidentally breaking something that used to work. The same risk exists for AI applications, but the "code" that can silently break is broader: a prompt edit, a system-message rewording, a schema change, or a model version bump can all degrade behavior in ways ordinary unit tests never touch, because unit tests (Lessons 1-5) intentionally use fakes and never exercise the real model's judgment at all.

This creates a real gap. Your unit tests can pass at 100% right after you change a prompt from "Summarize the article" to "Summarize the article in a formal tone," because the fake client returns the same canned text either way — but the actual summaries the real model produces might have gotten measurably worse for edge-case inputs. Regression testing at the evaluation layer is what catches this category of problem, and it does so by reusing the exact tools built in Lessons 6 and 7: a dataset, and a set of metrics, run consistently over time.

The Core Idea: A Baseline to Compare Against

A regression test for prompts or models works by running the same evaluation dataset through two versions of the system — a known-good baseline and the candidate you are considering shipping — and checking whether the candidate's metrics are no worse than the baseline's, within an acceptable tolerance.

def compare_against_baseline(baseline_metrics: dict, candidate_metrics: dict,
                              accuracy_tolerance: float = 0.02,
                              failure_rate_tolerance: float = 0.02) -> dict:
    accuracy_drop = baseline_metrics["accuracy"] - candidate_metrics["accuracy"]
    failure_rate_increase = candidate_metrics["failure_rate"] - baseline_metrics["failure_rate"]

    passed = (
        accuracy_drop <= accuracy_tolerance
        and failure_rate_increase <= failure_rate_tolerance
    )

    return {
        "passed": passed,
        "accuracy_drop": accuracy_drop,
        "failure_rate_increase": failure_rate_increase,
    }


def test_compare_against_baseline_passes_within_tolerance():
    baseline = {"accuracy": 0.90, "failure_rate": 0.02}
    candidate = {"accuracy": 0.885, "failure_rate": 0.03}

    result = compare_against_baseline(baseline, candidate)

    assert result["passed"] is True
    print("PASS: compare_against_baseline accepts a small, tolerable regression")


def test_compare_against_baseline_fails_on_large_accuracy_drop():
    baseline = {"accuracy": 0.90, "failure_rate": 0.02}
    candidate = {"accuracy": 0.70, "failure_rate": 0.02}

    result = compare_against_baseline(baseline, candidate)

    assert result["passed"] is False
    assert round(result["accuracy_drop"], 2) == 0.20
    print("PASS: compare_against_baseline flags a large accuracy regression")

The accuracy_tolerance and failure_rate_tolerance parameters exist because requiring an exact match to the baseline would be unrealistically strict — some variance between runs is expected even without any intentional change, due to the consistency effects covered in the previous lesson. Setting the tolerance too tight produces false alarms on every run; setting it too loose lets real regressions slip through unnoticed. A reasonable starting tolerance is informed directly by the consistency measurement from Lesson 7: if a system's baseline consistency score across repeated runs is 96%, a tolerance tighter than roughly that natural variance will fail spuriously on unchanged code.

Detecting Which Specific Examples Regressed

An aggregate pass/fail comparison tells you that something regressed, but not what. Because every dataset example carries a stable id (Lesson 6), you can compare per-example correctness between baseline and candidate runs directly, which is far more actionable during debugging.

def find_regressed_examples(baseline_results: dict, candidate_results: dict) -> list[str]:
    """Each *_results dict maps example_id -> bool (was the prediction correct)."""
    regressed = []
    for example_id, was_correct in baseline_results.items():
        if was_correct and not candidate_results.get(example_id, False):
            regressed.append(example_id)
    return regressed


def test_find_regressed_examples_identifies_newly_failing_cases():
    baseline_results = {
        "triage-001": True,
        "triage-002": True,
        "triage-003": False,
    }
    candidate_results = {
        "triage-001": True,
        "triage-002": False,  # this one used to pass
        "triage-003": False,
    }

    regressed = find_regressed_examples(baseline_results, candidate_results)

    assert regressed == ["triage-002"]
    print("PASS: find_regressed_examples pinpoints examples that newly started failing")

find_regressed_examples specifically looks for examples that flipped from correct to incorrect — it deliberately ignores examples that were already failing in the baseline, because those are pre-existing issues, not new regressions caused by the change under test. This distinction matters in practice: a prompt change might fix three previously-failing examples while breaking one that used to pass, and an aggregate accuracy score could show a net improvement while still hiding a real, specific regression worth investigating before shipping.

Structuring a Regression Test as a CI-Style Check

Bringing this together into something that runs automatically (for example, in a continuous integration pipeline whenever a prompt file or model configuration changes) means writing a test that fails loudly — with a non-zero exit code or a failed assertion — when a regression is detected, rather than one that just prints a report someone has to remember to read.

import json


def load_baseline_metrics(path: str) -> dict:
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)


def save_baseline_metrics(path: str, metrics: dict) -> None:
    with open(path, "w", encoding="utf-8") as f:
        json.dump(metrics, f, indent=2)


def test_prompt_change_does_not_regress_accuracy(tmp_path):
    baseline_path = tmp_path / "baseline_metrics.json"
    save_baseline_metrics(str(baseline_path), {"accuracy": 0.90, "failure_rate": 0.02})

    baseline_metrics = load_baseline_metrics(str(baseline_path))
    # In a real pipeline, candidate_metrics comes from actually running the
    # candidate prompt/model against the evaluation dataset with real API calls.
    candidate_metrics = {"accuracy": 0.91, "failure_rate": 0.015}

    comparison = compare_against_baseline(baseline_metrics, candidate_metrics)

    assert comparison["passed"], (
        f"Regression detected: accuracy dropped by {comparison['accuracy_drop']:.3f}, "
        f"failure rate increased by {comparison['failure_rate_increase']:.3f}"
    )
    print("PASS: candidate prompt/model does not regress against the stored baseline")

tmp_path here is a built-in pytest fixture that provides a fresh temporary directory unique to this test, automatically cleaned up afterward — a convenient way to test file-reading/writing code like load_baseline_metrics and save_baseline_metrics without leaving stray files behind or interfering with other tests. Note the comment marking where a real evaluation run against the actual candidate prompt or model would happen — this test's structure separates the comparison logic (fully unit-testable, as shown here) from the evaluation run itself (which does need a real or recorded API call, and is typically executed as a separate, less frequent step, then fed into this comparison).

Pinning Model Versions Deliberately

A regression can be introduced without anyone touching a prompt at all, simply because the underlying model was updated. This is why production systems generally pin an exact model identifier rather than relying on a generic alias that could point to a newer model version without warning.

# Preferred: an explicit, pinned model identifier.
STABLE_MODEL = "gpt-5.6-terra"

# Risky in production: an alias that may silently point to a different
# underlying model version over time, invalidating your regression baseline.
FLOATING_MODEL_ALIAS = "gpt-5.6-latest"

Note: Whether an alias like "gpt-5.6-latest" exists, and how it behaves, depends on what OpenAI offers at any given time — always check current model-naming documentation before relying on an alias in a production system, since this exact behavior is one of the details most likely to change between the time this is written and when you read it.

Pinning the model matters specifically because a regression test's baseline is only valid for a specific model version — if the model silently changes underneath a floating alias, your "regression" might actually be entirely explained by an upstream model update rather than by your own prompt change, and the two causes need to be distinguishable to fix the right thing.

When to Re-Baseline

A regression test's baseline is not meant to be permanent. When you intentionally accept a trade-off — for example, a slightly lower accuracy in exchange for a meaningfully lower failure rate, or a deliberate prompt rewrite that changes expected behavior — the baseline should be explicitly updated (and the update itself reviewed, ideally alongside the code change, the same way you would review a change to any other test's expected values) rather than left stale, or the regression test will keep failing for a difference you have already accepted.

Common Mistakes

• Comparing against no baseline at all, only a fixed threshold. A fixed "accuracy must be above 85%" check does not tell you whether a specific change made things better or worse; it can pass right through a real regression as long as the number stays above the line, and it can also block a legitimate improvement that happens to still be below the threshold for unrelated reasons.
• Setting tolerance to zero. Because model outputs have natural run-to-run variance (Lesson 7), a zero-tolerance comparison will frequently fail on unchanged code, training the team to ignore regression test failures altogether.
• Letting the model float via an unpinned alias. This makes it impossible to tell whether a regression came from your prompt change or from an unannounced model update, and it makes baselines unreliable over time.

Best Practices

• Store baseline metrics (and per-example results) somewhere versioned, so every prompt or model change under review can be compared against a known, agreed-upon prior state.
• Set tolerance thresholds informed by measured consistency, not by guesswork, so the regression test is sensitive enough to catch real problems without producing constant false alarms.
• Pin exact model versions in production and in regression baselines, and treat an intentional model upgrade as its own reviewed change with its own fresh baseline comparison, not a silent background shift.