Regression Testing for Prompt Changes
What Is a Regression? A regression is a case that worked before a change and fails after it. In software, regressions happen when fixing one bug breaks something else. Prompts are especially prone to them, because a prompt's behavior is shaped by every word, and a change aimed at one problem can quietly affect many other inputs.
Example A support classifier mislabels "Payment went through but I can't open the course" as Billing. You add a rule: "If the customer cannot access a course they paid for, choose Access." The case is fixed. But now "I was charged twice and can't see the second receipt" is also labeled Access, because it mentions both payment and not being able to see something. Overall accuracy may even look similar, but the prompt has changed behavior on cases that used to be right.
Regression testing means re-running a fixed set of test cases after every change and comparing the results case by case with the results before the change.
What Triggers Regression Tests Regressions are not caused only by editing the prompt text.
Change Example Prompt text A new rule, a reworded instruction, a new example Shared fragments A change to a data-handling rule included in many prompts (Lesson 68) Model A new model version or a different model (Lesson 67) Settings Temperature, output limits, reasoning settings Retrieval New chunking, new embedding model, updated documents (Lesson 101) Tools Changed tool descriptions or result formats (Lesson 109) Code around the prompt New validation rules, parsing changes Any of these should trigger the regression suite before the change reaches users.
The Regression Testing Process
- Keep a stable regression suite: test cases with expected results (Lesson 115)
- Record a baseline: the results of the current version on every case
- Make the change
- Run the same suite on the new version
- Compare case by case: fixed, broken, still failing, unchanged
- Apply a gate: pass, review, or block
- If accepted, the new results become the baseline Why Case-by-Case Comparison Matters Overall scores can hide regressions. If a change fixes two cases and breaks two others, the total stays the same, but the two broken cases may be the important ones. Comparing individual cases reveals exactly what changed (Lesson 66, paired comparison).
Comparing Runs in Code def compare_runs(baseline: dict[str, bool], candidate: dict[str, bool], critical_ids: frozenset[str] = frozenset()) -> tuple[dict, float, float, str]: """Compare pass/fail results per case and decide whether the change may proceed.""" report = {"fixed": [], "broken": [], "still_failing": [], "missing_in_candidate": []} for case_id, passed_before in baseline.items(): if case_id not in candidate: report["missing_in_candidate"].append(case_id) continue passed_now = candidate[case_id] if passed_now and not passed_before: report["fixed"].append(case_id) elif passed_before and not passed_now: report["broken"].append(case_id) elif not passed_before and not passed_now: report["still_failing"].append(case_id)
before = sum(baseline.values()) / len(baseline) after = sum(candidate.get(case_id, False) for case_id in baseline) / len(baseline) critical_broken = [case_id for case_id in report["broken"] if case_id in critical_ids]
if critical_broken or report["missing_in_candidate"] or after < before: decision = "block" elif report["broken"]: decision = "review" else: decision = "pass" return report, round(before, 2), round(after, 2), decision
baseline = {"t1": True, "t2": True, "t3": False, "t4": True, "t5": False, "t6": True} candidate = {"t1": True, "t2": False, "t3": True, "t4": True, "t5": True, "t6": True}
print(compare_runs(baseline, candidate)) print(compare_runs(baseline, candidate, critical_ids=frozenset({"t2"}))[3]) print(compare_runs(baseline, {**candidate, "t2": True})[3]) Output:
({'fixed': ['t3', 't5'], 'broken': ['t2'], 'still_failing': [], 'missing_in_candidate': []}, 0.67, 0.83, 'review') block pass Code Explanation Categories Each case is compared between the baseline and the candidate (the new version):
fixed: failed before, passes now broken: passed before, fails now (a regression) still_failing: failed in both missing_in_candidate: not run in the new version, which means the comparison is incomplete Pass rates before and after are the share of cases passing. The candidate improves the pass rate from 0.67 to 0.83.
The gate
block if a critical case broke, if any case was not run, or if the overall pass rate dropped. review if any non-critical case broke: a person must look at it and decide. pass if nothing broke. Reading the Three Results The candidate fixes two cases but breaks t2. The overall rate improved, so the decision is review, not pass: someone must check whether breaking t2 is acceptable. When t2 is marked critical, for example because it represents a refund rule, the same results are blocked. When t2 also passes, nothing is broken and the change passes. This gate encodes an important principle: an improvement in the overall score does not justify breaking important cases without a deliberate decision.
Critical Cases Not all test cases are equally important. Mark as critical the cases where a failure would cause real harm:
Critical Case Type Example Policy and money Refund eligibility at the 7-day boundary Safety Prompt injection attempts that must be refused (Lessons 121–125) Privacy Requests for another student's data must be refused Format required by code Outputs consumed by automation must always parse Previously fixed bugs Every serious bug, once fixed, becomes a critical case Critical cases should never be broken without an explicit, documented decision.
Handling Inconsistent Results Model outputs can vary between runs (Lesson 120). A case that passes and fails at random would appear as "fixed" or "broken" by chance, creating false alarms or hiding real regressions.
Run Each Case Several Times For cases with variable outputs, run them several times and compare pass rates rather than single results:
Case Baseline Pass Rate Candidate Pass Rate Interpretation t7 5/5 5/5 Stable pass t8 5/5 2/5 Regression t9 3/5 3/5 Unstable in both: investigate the prompt, not the change Practical Rules Treat a case as passing only if it passes in all, or nearly all, of several runs. Define the threshold in advance. Cases that are unstable in the baseline are a problem in themselves; fix the instability before relying on them (Lesson 120). Use deterministic settings where your model and API support them, to reduce variation during testing. Running Regression Tests Automatically Regression tests are most effective when they run automatically, in the same way as code tests:
Pull request changes a prompt file ↓ CI runs the regression suite with the new prompt version ↓ compare_runs() against the stored baseline ↓ pass → merge allowed review → reviewer must approve broken cases with a note block → merge not allowed until fixed What to Store Item Why Prompt version and model for each run Know exactly what was tested (Lesson 67) Full outputs, not only pass/fail Read what changed in broken cases Per-case results Compare runs case by case The baseline used for comparison Reproduce the decision later Cost Control Regression suites call the model many times. Keep a core suite of the most important and representative cases that runs on every change, and run a larger suite before releases or model upgrades.
Regression Testing When Models Change Model updates can change behavior across many cases at once, even when your prompt is unchanged. Before switching models or versions:
Run the full regression suite on the new model. Compare case by case with the current model's baseline. Read every broken case, and check critical cases first. Adjust prompts if needed, and re-run. Switch only when the gate passes, and keep the old model available for rollback. Pinning model versions in production, where your provider allows it, ensures that model changes happen when you choose, after testing (Lesson 67).
Common Mistakes Mistake 1: Comparing Only Overall Scores What it is: Accepting a change because the average improved.
Why it happens: Averages are easy to compare.
What it causes: Important cases break unnoticed.
How to fix it: Compare case by case and review every broken case.
Mistake 2: No Stable Suite What it is: Testing each change on a different set of examples.
Why it happens: New examples are chosen for each change.
What it causes: Regressions in older cases are never checked.
How to fix it: Keep a stable regression suite, and add to it rather than replacing it.
Mistake 3: Ignoring Non-Prompt Changes What it is: Running regression tests only when prompt text changes.
Why it happens: The prompt seems to be the only thing that affects behavior.
What it causes: Model, retrieval, and tool changes introduce regressions silently.
How to fix it: Trigger the suite for every change that can affect outputs.
Mistake 4: Treating Random Variation as Regression What it is: Reacting to single-run differences on unstable cases.
Why it happens: Each case is run once.
What it causes: False alarms, or real regressions dismissed as noise.
How to fix it: Run unstable cases several times and compare pass rates.
Mistake 5: Updating Expectations to Make Tests Pass What it is: Changing a test's expected result because the new version fails it.
Why it happens: It makes the gate pass.
What it causes: Regression tests stop protecting anything.
How to fix it: Change expectations only when the requirement itself changed, with a documented reason.
Best Practices Keep a stable regression suite. Include critical cases and every fixed bug.
Store baselines with versions and full outputs. Make comparisons reproducible.
Compare case by case. Fixed, broken, still failing, missing.
Gate changes automatically. Block critical breaks; require review for others.
Run the suite for every change that affects outputs. Prompts, fragments, models, settings, retrieval, tools.
Handle variation deliberately. Multiple runs and pass-rate thresholds for unstable cases.
Try It Yourself Exercise 1: Baseline and Candidate Run your test suite from Lesson 115 with your current prompt and save the pass/fail result for each case. Make one prompt change, run the suite again, and use compare_runs() to produce a report and a decision.
Exercise 2: Critical Cases Mark at least five cases in your suite as critical, with a one-sentence reason for each. Make a deliberately harmful prompt change, such as removing the boundary rule, and confirm that the gate blocks it.
Exercise 3: Unstable Cases Run every case in your suite five times. Identify cases that do not pass consistently in the baseline, and list them. For each, decide whether the instability comes from an ambiguous test case or from the prompt, and fix one of them.