Recalling the Basics, Applying Them to New Ground

Unit 13, Lesson 2 introduced the core anatomy of an evaluation dataset: a collection of inputs, each paired with an expected output or grading criteria, used to measure how well a model-driven task performs. That lesson built a dataset around grading free-form model responses. This lesson applies the same underlying idea to a different, equally common situation: evaluating a classification-style task — one where the model's job is to pick a label from a fixed set — which is exactly the kind of task that shows up constantly in real applications (support ticket routing, content moderation, intent detection) and which pairs naturally with the tool-calling and structured-output testing from the last two lessons.

The dataset-building principles carry over directly: realistic inputs, a clear notion of "correct," enough examples to be meaningful, and deliberate coverage of edge cases — but the shape of the dataset and the way it gets used differs enough from Unit 13's example to be worth building from scratch here.

Defining the Task Precisely

Before writing a single example, the task needs an unambiguous definition, because an evaluation dataset is only as good as the labels in it. Consider a support-ticket triage task: given the text of a customer message, classify it into exactly one of billing, technical, account, or other.

VALID_CATEGORIES = {"billing", "technical", "account", "other"}


def is_valid_category(label: str) -> bool:
    return label in VALID_CATEGORIES

This tiny function is worth writing and testing on its own, independent of the model: it defines the boundary of what counts as a valid answer at all, and every dataset example and every grading step will rely on it. Skipping this step and leaving the category set implicit is a common source of disagreement later, when two people labeling data disagree about whether a billing question that also mentions a login problem is billing or account.

Structuring the Dataset

A small evaluation dataset is typically stored as a list of records, each with an input and the expected label, often persisted as JSON Lines (.jsonl) — one JSON object per line — because it is easy to append to, easy to diff in version control, and easy to stream without loading the whole file into memory.

import json

TRIAGE_DATASET = [
    {
        "id": "triage-001",
        "input": "I was charged twice for my subscription this month, can you refund one?",
        "expected_category": "billing",
    },
    {
        "id": "triage-002",
        "input": "The app crashes every time I try to upload a photo larger than 5MB.",
        "expected_category": "technical",
    },
    {
        "id": "triage-003",
        "input": "I can't log in anymore, it says my email isn't recognized.",
        "expected_category": "account",
    },
    {
        "id": "triage-004",
        "input": "Do you have a mobile app for Android?",
        "expected_category": "other",
    },
    {
        "id": "triage-005",
        "input": "My card on file expired and now I can't access premium features.",
        "expected_category": "billing",
    },
]


def save_dataset_as_jsonl(dataset: list[dict], path: str) -> None:
    with open(path, "w", encoding="utf-8") as f:
        for record in dataset:
            f.write(json.dumps(record) + "\n")


def load_dataset_from_jsonl(path: str) -> list[dict]:
    records = []
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                records.append(json.loads(line))
    return records

Each record carries a stable id, which matters more than it might first appear: an id lets you track a specific example's pass/fail status across runs and across model or prompt versions (used heavily in the regression-testing lesson later in this unit), rather than only knowing an aggregate pass rate that could hide the fact that the same three examples fail every single time. save_dataset_as_jsonl and load_dataset_from_jsonl are ordinary, fully deterministic functions — worth unit testing themselves, using the fakes-and-mocks-free techniques from Lesson 1, since a bug in how the dataset is read or written silently corrupts every evaluation built on top of it.

Sizing a Small Dataset and Deciding What "Small" Means

A dataset with five examples, as shown above, is enough to illustrate structure and mechanics, but not enough to draw a reliable conclusion about real-world accuracy — five examples can be memorized by chance or misjudged by an unlucky sample. A genuinely useful small evaluation dataset for a task like ticket triage typically starts in the range of 30 to 100 examples: large enough that a percentage score (like 92% accuracy) is statistically meaningful rather than noise from one or two lucky or unlucky examples, but small enough to build and review by hand without requiring a data-labeling team.

The right size depends on how many distinct behaviors you need to distinguish. If the categories are well separated (a billing complaint reads very differently from a technical bug report), fewer examples are needed to get a stable signal than if the categories overlap heavily (a message that is both a login problem and a billing complaint). As a rule of thumb, deliberately include several examples of the hardest real cases — the ones that could plausibly belong to more than one category — because those are exactly the cases where a model's actual behavior diverges most from what a naive test would reveal.

Covering Edge Cases Deliberately

A dataset built only from clean, obvious examples measures the easiest 80% of the problem and tells you almost nothing about the hard 20% that causes real support escalations. Building in edge cases on purpose is what turns a dataset from a demo into a useful tool.

EDGE_CASE_EXAMPLES = [
    {
        "id": "triage-edge-001",
        "input": "",  # empty input
        "expected_category": "other",
    },
    {
        "id": "triage-edge-002",
        "input": "I can't pay my bill because I can't log in to update my card.",
        "expected_category": "account",  # ambiguous: billing + account, but the blocker is login
    },
    {
        "id": "triage-edge-003",
        "input": "asdkjaslkdj",  # gibberish / non-language input
        "expected_category": "other",
    },
    {
        "id": "triage-edge-004",
        "input": "Please cancel my account and stop billing me immediately.",
        "expected_category": "billing",  # cancellation framed as a billing action
    },
]

FULL_TRIAGE_DATASET = TRIAGE_DATASET + EDGE_CASE_EXAMPLES

triage-edge-002 is deliberately ambiguous — a reasonable person could argue for either billing or account — and the dataset resolves that ambiguity explicitly by picking one label and documenting the reasoning in a comment. This is an important discipline: every ambiguous example in a real dataset should have its resolution justified somewhere (a comment, a labeling guideline document, or a note field), because without that, disagreements resurface every time someone reviews the dataset later, and the dataset's "correct" answers start to look arbitrary.

triage-edge-001 (empty input) and triage-edge-003 (gibberish) test something different from category-boundary ambiguity: they test whether the task handles degenerate input gracefully at all, which matters because real user input is messier than any curated dataset naturally suggests.

Splitting Dataset-Building From Grading

Notice that nothing in this dataset itself calls the model or computes a score — it is pure data. This separation is intentional and mirrors the design principle from earlier lessons: keep deterministic, testable pieces (the dataset structure, the loading/saving code, the validity check) separate from the piece that involves the model. The next lesson uses this exact dataset to compute accuracy, consistency, and failure-rate metrics; keeping the dataset itself free of any model-calling logic means it can be unit tested, version controlled, and reused across many different evaluation runs — including regression tests that compare two different prompts or model versions against the identical set of examples, which only works cleanly if the dataset does not change between runs.

Common Mistakes

• Building a dataset with only easy, unambiguous examples. A model can score 100% on such a dataset while still failing regularly in production, because production input distribution always includes messier cases than a hastily assembled dataset does.
• Leaving ambiguous labels undocumented. When two team members disagree about what the "correct" label should have been for an ambiguous example, and there is no recorded justification, the dataset's reliability as a ground truth erodes over time.
• Conflating the dataset with the grading logic. Embedding scoring logic directly alongside the data (for example, hardcoding a comparison function next to each record) makes the dataset harder to reuse across different evaluation approaches, such as exact-match grading versus a model-graded rubric.

Best Practices

• Give every example a stable, unique ID so results can be tracked per example across multiple evaluation runs, not just as an aggregate score.
• Deliberately include edge cases and ambiguous examples, and document why each one was assigned its expected label, rather than relying only on clear-cut cases.
• Keep the dataset as plain, model-agnostic data (for example, JSON Lines) separate from any grading or model-calling code, so the same dataset can support multiple different evaluation runs, prompts, or model versions.