Testing Prompts Against Representative Datasets

Unit 13 introduced how to build and grade an eval: assembling a dataset of representative inputs, defining a grading method, and running a model against the dataset to produce a score. This lesson applies that methodology to a specific, recurring engineering task raised repeatedly throughout this unit — deciding whether a prompt change (a new version, per Lesson 8; a tightened output requirement, per Lesson 7; a different set of few-shot examples, per Lesson 4) actually improved behavior, rather than guessing from a handful of manual spot checks.

Why Manual Spot Checks Are Not Enough

The natural way to check whether a prompt edit helped is to try it on two or three examples and read the output. This is a reasonable first pass during active editing, but it does not scale to a reliable engineering decision, for a specific reason: prompt changes rarely affect every input uniformly. A change that fixes a formatting issue on typical inputs might introduce a regression only on inputs with unusual structure, and a handful of manually chosen test cases — usually the easy, typical ones that come to mind first — are exactly the cases least likely to reveal that kind of regression. A representative dataset, run systematically, is what surfaces it.

Building a Representative Dataset

A dataset for prompt testing is a collection of realistic inputs paired with either an expected output or a way to grade the actual output's quality. It should reflect the diversity of real usage, not just the typical case:

from dataclasses import dataclass

@dataclass(frozen=True)
class TestCase:
    input_text: str
    expected_category: str

TICKET_TEST_CASES: list[TestCase] = [
    TestCase("I was charged twice this month.", "billing"),
    TestCase("The app crashes every time I upload a photo.", "technical"),
    TestCase("How do I change my account email?", "account"),
    TestCase("Can you recommend a good restaurant nearby?", "other"),
    TestCase("", "other"),  # edge case: empty input
    TestCase("My invoice shows a charge but the app also won't load.", "billing"),  # ambiguous, multi-topic
]

Three properties make this dataset useful rather than decorative. First, it includes an edge case (empty input) that a developer testing informally would likely never think to try, but that real production traffic will eventually send. Second, it includes an intentionally ambiguous case (a ticket that touches both billing and a technical symptom) precisely because ambiguous cases are where prompt changes most often cause visible behavior shifts — a change to the category list or the instructions wording can flip how these borderline cases resolve, even while easy cases stay stable. Third, each case's expected_category is a genuine judgment call the dataset's author made deliberately, which is itself worth documenting — for the ambiguous case above, "billing" was chosen because the invoice complaint was mentioned first and is the more actionable-sounding half.

Note: Sourcing test cases from real, anonymized production inputs (with appropriate handling of any sensitive data) generally produces a more representative dataset than inventing cases from imagination, for the same reason noted in Lesson 4 about few-shot examples — real usage patterns are harder to guess correctly than they are to observe.

Running a Prompt Against the Dataset

With a dataset defined, running the current prompt version against every case and grading the results follows directly from Unit 13's eval structure:

from openai import OpenAI

client = OpenAI()

def classify_ticket(ticket_text: str, instructions: str) -> str:
    response = client.responses.create(
        model="gpt-5.6-terra",
        instructions=instructions,
        input=ticket_text if ticket_text else "(empty message)",
    )
    return response.output_text.strip().lower()

def run_eval(test_cases: list[TestCase], instructions: str) -> dict:
    results = []
    correct = 0
    for case in test_cases:
        predicted = classify_ticket(case.input_text, instructions)
        is_correct = predicted == case.expected_category
        correct += is_correct
        results.append({
            "input": case.input_text,
            "expected": case.expected_category,
            "predicted": predicted,
            "correct": is_correct,
        })
    return {
        "accuracy": correct / len(test_cases),
        "results": results,
    }

run_eval returns both a single summary number (accuracy) and the per-case detail (results). The summary number is what you track over time and compare across prompt versions; the per-case detail is what you actually read when accuracy drops, because a single number cannot tell you which cases regressed or why — only the individual results can.

Comparing Two Prompt Versions

The direct application of this to Lesson 8's versioning: run the same dataset against both the current and candidate prompt versions, and compare accuracy and, critically, which specific cases changed:

def compare_prompt_versions(test_cases: list[TestCase], old_instructions: str, new_instructions: str) -> dict:
    old_eval = run_eval(test_cases, old_instructions)
    new_eval = run_eval(test_cases, new_instructions)

    regressions = []
    improvements = []
    for old_r, new_r in zip(old_eval["results"], new_eval["results"]):
        if old_r["correct"] and not new_r["correct"]:
            regressions.append(new_r)
        elif not old_r["correct"] and new_r["correct"]:
            improvements.append(new_r)

    return {
        "old_accuracy": old_eval["accuracy"],
        "new_accuracy": new_eval["accuracy"],
        "regressions": regressions,
        "improvements": improvements,
    }

OLD_INSTRUCTIONS = "Classify the ticket into billing, technical, account, or other. Respond with only the category."
NEW_INSTRUCTIONS = (
    "Classify the support ticket into exactly one of: billing, technical, "
    "account, other. If the ticket mentions multiple topics, choose the "
    "one that seems most urgent or actionable. Respond with only the "
    "category name, lowercase, no punctuation."
)

comparison = compare_prompt_versions(TICKET_TEST_CASES, OLD_INSTRUCTIONS, NEW_INSTRUCTIONS)
print(f"Old accuracy: {comparison['old_accuracy']:.2f}")
print(f"New accuracy: {comparison['new_accuracy']:.2f}")
print(f"Regressions: {len(comparison['regressions'])}")
print(f"Improvements: {len(comparison['improvements'])}")

The regressions and improvements lists are more decision-relevant than the two accuracy numbers alone. A new prompt version that raises overall accuracy from 0.80 to 0.83 sounds like an unambiguous win until you check regressions and find that it broke a case your team considers especially important — an accuracy improvement that trades away correctness on a high-value case is not automatically a good trade, and only inspecting the per-case detail reveals that tradeoff exists at all.

Grading Tasks Without an Exact Expected Output

Classification has a clean notion of "correct" — an exact string match. Summarization and transformation (Lesson 6) do not, since there is no single correct summary to match against. For these, grading needs either a rubric checked by a separate model call (a technique Unit 13 refers to as model-graded evaluation) or a set of mechanical property checks, as introduced in Lesson 7's check_output_shape:

@dataclass(frozen=True)
class SummaryTestCase:
    document: str
    max_sentences: int

SUMMARY_TEST_CASES = [
    SummaryTestCase("Long article text about a product launch...", 3),
    SummaryTestCase("Another article about a company earnings report...", 3),
]

def evaluate_summary_properties(summary: str, max_sentences: int) -> dict:
    sentence_count = summary.count(".") + summary.count("!") + summary.count("?")
    return {
        "within_length": sentence_count <= max_sentences,
        "non_empty": len(summary.strip()) > 0,
        "sentence_count": sentence_count,
    }

This does not verify that the summary is a good summary — that still requires either human review or a model-graded rubric, both covered in Unit 13 — but it verifies the mechanical properties (length, non-emptiness) cheaply and deterministically across every case in the dataset, catching a meaningful class of regressions (a prompt change that starts producing five-sentence summaries instead of three) without needing a second model call per case.

Building a Regression Test Suite From Prior Failures

A dataset should grow over time, specifically by adding every real failure encountered in production as a new permanent test case, so the same mistake cannot silently reappear in a future prompt version without being caught:

def add_regression_case(test_cases: list[TestCase], input_text: str, correct_category: str) -> list[TestCase]:
    return test_cases + [TestCase(input_text, correct_category)]

# A real misclassification found in production, on 2027-02-10:
# ticket was routed to "other" but should have been "technical"
TICKET_TEST_CASES = add_regression_case(
    TICKET_TEST_CASES,
    "Nothing happens when I click the export button.",
    "technical",
)

This is the same principle as regression testing in ordinary software: a bug found in production becomes a permanent test case precisely because prompt behavior is not guaranteed to stay stable across future edits, few-shot example changes, or even model version upgrades — the case that broke once can break again in a future change unless something is actively checking for it every time.

Testing the Evaluation Logic Itself

The comparison and grading functions are themselves plain Python and should be tested directly, using fake predictions rather than live model calls, to make sure the evaluation harness is trustworthy before relying on its verdicts:

def test_run_eval_computes_accuracy_correctly():
    def fake_classify(ticket_text: str, instructions: str) -> str:
        return "billing" if "charged" in ticket_text else "other"

    cases = [
        TestCase("I was charged twice.", "billing"),
        TestCase("Tell me a joke.", "other"),
        TestCase("I was charged incorrectly.", "technical"),  # will mismatch
    ]
    correct = sum(1 for c in cases if fake_classify(c.input_text, "") == c.expected_category)
    accuracy = correct / len(cases)
    assert abs(accuracy - (2 / 3)) < 1e-9
    print("PASS: accuracy computed correctly against known fake predictions")

def test_compare_detects_regression():
    old_results = [{"correct": True}, {"correct": True}]
    new_results = [{"correct": True}, {"correct": False}]
    regressions = [n for o, n in zip(old_results, new_results) if o["correct"] and not n["correct"]]
    assert len(regressions) == 1
    print("PASS: comparison correctly identifies a single regression")

test_run_eval_computes_accuracy_correctly()
test_compare_detects_regression()

Common Mistakes

Testing a prompt change against only a few manually chosen, typical examples. This reliably misses regressions on edge cases and ambiguous inputs, which is exactly where prompt wording changes tend to shift behavior the most.

Comparing only aggregate accuracy between prompt versions, without inspecting per-case regressions. An improved overall score can mask the loss of a specific, high-value case; always check which individual cases flipped, not just the summary number.

Never adding production failures back into the test dataset. Without this feedback loop, the same mistake can resurface silently in a later prompt version, because nothing in the test suite would catch it a second time.

Best Practices

Build the test dataset to include edge cases and ambiguous inputs deliberately, not just typical ones. These are the cases most likely to reveal a regression when a prompt changes.

Compare prompt versions on the same fixed dataset and inspect both aggregate accuracy and per-case regressions before rolling out a change. Follow the gradual rollout approach from Lesson 8 once the comparison looks favorable.

Turn every real production failure into a permanent regression test case. This is the mechanism that keeps a growing prompt codebase from repeating its own past mistakes as it evolves.