Building a Prompt Evaluation Dataset
From Test Cases to a Dataset Lesson 115 explained how to write individual test cases. An evaluation dataset is a larger, deliberately designed collection of test cases that represents the inputs your prompt will face in real use. It is the foundation of reliable evaluation (Lesson 114), optimization (Lesson 66), and regression testing (Lesson 118).
A small, hand-written set of test cases is a good start. But as a prompt becomes important to your product, the questions change:
Do my test cases reflect what real users actually send? Are all important categories represented, including rare ones? Are the expected answers correct and consistent? Am I accidentally testing on the same examples I used to build the prompt? Is the dataset still accurate after policies changed? This lesson covers how to build a dataset that answers these questions well.
The Dataset Building Process
- Define what the dataset must represent
- Collect candidate inputs (real data, failures, written, generated)
- Remove personal and sensitive information
- Sample for balance and coverage
- Label with clear guidelines, and check label agreement
- Split into development and test sets
- Check for leakage into prompts
- Version, document, and maintain Step 1: Define What the Dataset Must Represent Write down the scope before collecting anything:
Question Example Answer for a Support Classifier Which task? Classifying student support messages into 4 categories Which inputs? Messages from the website chat and email, in English, Tamil, and mixed languages Which categories must be covered? All 4 labels, plus empty, very long, and adversarial messages What counts as correct? The label defined in the labeling guidelines What size is needed? At least 30 cases per label in the test set The scope prevents a common problem: a dataset that is easy to build but does not match the real use case.
Step 2: Collect Candidate Inputs Source Strength Caution Real inputs (logs, tickets, chats) Most realistic wording and distribution Must be anonymized; may lack rare cases Production failures Target known weaknesses Bias toward past problems Written by the team Cover rules, boundaries, and rare cases deliberately Often cleaner than real inputs Generated by a model Fast; good for edge and adversarial variety (Lesson 88) Must be reviewed; can share the prompt's blind spots A strong dataset combines all four, with real inputs as the foundation.
Step 3: Remove Personal and Sensitive Information Real inputs contain names, emails, phone numbers, order IDs, and sometimes much more. Evaluation datasets are stored for a long time and shared with team members and tools, so personal data must be removed before inputs are added.
Replace personal details with realistic placeholders, such as asha@example.com (Lesson 88). Use code for structured patterns and review for the rest (Lesson 81). Follow your organization's data policies about what may be used for evaluation at all. Step 4: Sample for Balance and Coverage Real data is imbalanced: most messages may be about billing, with few about technical problems. If you sample randomly, rare categories may be almost absent, and the dataset cannot tell you how the prompt handles them.
Stratified sampling solves this: divide inputs into groups, called strata, such as categories, and sample from each group separately.
import hashlib import random from collections import Counter, defaultdict
def stratified_sample(items: list[dict], key: str, per_group: int, seed: int = 42) -> list[dict]: """Take up to per_group items from each group, reproducibly.""" groups = defaultdict(list) for item in items: groups[item[key]].append(item) rng = random.Random(seed) sample = [] for group in sorted(groups): members = groups[group][:] rng.shuffle(members) sample.extend(members[:per_group]) return sample
logs = ([{"id": f"b{i}", "category": "Billing", "input": f"billing message {i}"} for i in range(60)]
- [{"id": f"a{i}", "category": "Access", "input": f"access message {i}"} for i in range(25)]
- [{"id": f"t{i}", "category": "Technical", "input": f"technical message {i}"} for i in range(8)])
sample = stratified_sample(logs, "category", per_group=10) print(Counter(item["category"] for item in sample)) Output:
Counter({'Access': 10, 'Billing': 10, 'Technical': 8}) Code Explanation Items are grouped by the key field, here the category. Each group is shuffled with a seeded random generator (random.Random(seed)), so the same sample is produced every time, which makes the dataset reproducible. Up to per_group items are taken from each group. Billing had 60 candidates but contributes only 10; Technical had only 8, so all 8 are used. Reading the Result The sample is balanced for evaluation, unlike the original logs, where Billing outnumbered Technical by more than 7 to 1. The Technical group has only 8 items, which shows where more data is needed: collect more real technical messages or write additional cases.
Balanced vs Representative Dataset Type Use For Balanced (equal per category) Measuring how well each category is handled Representative (same proportions as real traffic) Estimating the overall accuracy users will experience Many teams keep both, or report per-category results from a balanced set alongside an estimate weighted by real traffic proportions.
Step 5: Label With Guidelines and Check Agreement Labeling Guidelines Labeling guidelines are written instructions that tell people how to assign expected results. They include:
The definition of each label Rules for boundary cases (Lesson 18) Examples of difficult cases, with the correct label and the reason What to do when a case is truly ambiguous, such as marking it for discussion The same guidelines should match the rules in the prompt. If the prompt and the dataset disagree on a rule, evaluation results measure the disagreement, not prompt quality.
Measure Agreement Between Labelers Have two people label a sample independently and compare their labels. Low agreement on a category means the definition or guidelines are unclear, and the "correct" answer is uncertain. Fix the guidelines before trusting results for that category. The agreement measures from Lesson 116 apply here as well.
Step 6: Split Into Development and Test Sets As explained in Lessons 64 and 66, use separate sets for different purposes:
Set Purpose Development set Used repeatedly while writing and improving prompts Test set Used rarely, to measure final performance without bias Stable Splits With Hashing Splitting by hashing each case's ID keeps each case in the same set permanently, even as new cases are added (the same technique as in Lesson 69):
def assign_split(case_id: str, test_share: float = 0.3) -> str: bucket = int(hashlib.sha256(case_id.encode()).hexdigest()[:8], 16) / 0xFFFFFFFF return "test" if bucket < test_share else "dev"
print(Counter(assign_split(item["id"]) for item in sample)) print(assign_split("b3") == assign_split("b3")) Output:
Counter({'dev': 19, 'test': 9}) True Code Explanation The case ID is hashed and converted to a number between 0 and 1. Cases below test_share go to the test set; the rest go to the development set. The same ID always produces the same split, so adding new cases never moves existing cases between sets. With small datasets, the actual proportion varies from the target (here 9 of 28, about 32%, close to the 30% target).
Step 7: Check for Leakage Leakage happens when test cases, or close copies of them, appear in the prompt itself, typically as few-shot examples (Lesson 26). The prompt then "knows" the answers to those test cases, and evaluation results are too optimistic.
def find_leakage(examples: list[str], test_cases: list[dict]) -> list[str]: """Return ids of test cases whose input also appears among the prompt's examples.""" normalize = lambda text: " ".join(text.lower().split()) test_inputs = {normalize(case["input"]): case["id"] for case in test_cases} return [test_inputs[normalize(example)] for example in examples if normalize(example) in test_inputs]
test_cases = [item for item in sample if assign_split(item["id"]) == "test"] few_shot_examples = ["Billing message 3", "Some new example", test_cases[0]["input"].upper()] print(find_leakage(few_shot_examples, test_cases)) Output:
['a9'] Code Explanation Inputs are normalized by lowercasing and collapsing whitespace, so a copy in different capitalization is still detected. The third few-shot example is an uppercased copy of test case a9, and it is reported. "Billing message 3" is not reported because that case is in the development set, where using it as an example is acceptable. This check catches exact and near-exact copies only. Paraphrased copies require similarity checks (Lesson 27). Make it a rule: examples in prompts come from the development set or are written separately, never from the test set.
Step 8: Version, Document, and Maintain Version the Dataset Store datasets in version control or a data store with versions. Every evaluation result should record which dataset version it used (Lesson 67), so results can be compared fairly.
Document It A short dataset card describes the dataset:
Dataset: support-classifier-eval Version: 3 (2026-10-30) Scope: student support messages; English, Tamil, and mixed language Size: 240 cases (dev 168, test 72); 60 per label Sources: anonymized chat logs (70%), production failures (15%), written cases (15%) Labeling: two labelers, guidelines v2; disagreements resolved by the support lead Known gaps: few voice-transcribed messages Maintain It Event Action A policy changes Review and update affected expected results, with a note A production failure is found Add it as a new case A new feature or category is added Add cases for it User behavior changes Refresh samples from recent real inputs An outdated dataset is dangerous: a prompt that correctly follows a new policy will fail tests built on the old one.
Common Mistakes Mistake 1: Only Convenient Examples What it is: Building the dataset from examples that are easy to write.
Why it happens: Real data takes effort to collect and clean.
What it causes: High scores that do not reflect real performance.
How to fix it: Base the dataset on anonymized real inputs, supplemented by written and generated cases.
Mistake 2: Random Sampling of Imbalanced Data What it is: Sampling randomly from logs dominated by one category.
Why it happens: Random sampling seems fair.
What it causes: Rare categories are barely tested.
How to fix it: Use stratified sampling, and collect more data for small groups.
Mistake 3: Unclear Labels What it is: Labeling without written guidelines or agreement checks.
Why it happens: Labels seem obvious.
What it causes: Inconsistent expected results that make evaluation unreliable.
How to fix it: Write guidelines, double-label a sample, and resolve disagreements.
Mistake 4: Test Cases Used as Prompt Examples What it is: Copying test inputs into few-shot examples.
Why it happens: Good test cases look like good examples.
What it causes: Inflated test scores.
How to fix it: Keep a strict separation and check for leakage.
Mistake 5: Never Updating the Dataset What it is: Using the same dataset after policies and products change.
Why it happens: Datasets are treated as finished.
What it causes: Correct behavior is marked as wrong, and new situations are untested.
How to fix it: Version the dataset and review it when policies, features, or users change.
Best Practices Define the dataset's scope first. Task, inputs, categories, correctness, size.
Build on anonymized real inputs. Add failures, written cases, and reviewed generated cases.
Sample by category. Balance for per-category measurement; know the real proportions too.
Label with guidelines and check agreement. Align guidelines with the prompt's rules.
Split into stable development and test sets. And keep test cases out of prompts.
Version, document, and maintain the dataset. Treat it as an important project asset.
Try It Yourself Exercise 1: Plan a Dataset Write the scope for an evaluation dataset for one of your prompts: task, inputs, categories, definition of correctness, and target size. List which sources you will use and in what proportions.
Exercise 2: Sample and Split Collect or write at least 100 candidate inputs with categories. Use stratified_sample() to select a balanced sample and assign_split() to divide it. Report the size of each category in each split, and identify categories that need more data.
Exercise 3: Leakage and Documentation Check your prompt's few-shot examples against your test set with find_leakage(), and replace any leaked examples. Then write a dataset card for your dataset, including known gaps.