Timing Note

Before You Build: A Timing Note Worth Knowing

Before writing any code, you need one piece of context that changes how you should read this lesson: OpenAI has announced that the Evals platform — the dashboard and the API this lesson teaches — is being wound down. According to OpenAI's own deprecations page, existing evals become read-only on October 31, 2026, and the platform (both dashboard and API) is scheduled to shut down on November 30, 2026. OpenAI's migration guidance points existing users toward third-party tools such as Promptfoo for ongoing evaluation work going forward.

That doesn't make this lesson pointless — quite the opposite. The Evals API is still live today, it's still the clearest, most explicit teaching example of what an evaluation pipeline actually consists of, and every concept you learn here — a structured dataset, testing criteria that define "correct," a run that executes your prompt against that dataset, and a scored result you can compare over time — is the exact same shape you'll find in Promptfoo, in a hand-rolled evaluation harness, or in whatever OpenAI replaces this platform with. Learn the concepts here, where the API makes every piece explicit and nameable, and you'll be able to apply the same thinking wherever you end up running evaluations after the shutdown date. If you're reading this after November 30, 2026, treat the workflow below as the conceptual blueprint and swap in your evaluation tool of choice — the ideas transfer directly.

What the Evals API Actually Does

An eval, in the OpenAI API, is a named, reusable configuration made of two things:

1. A data source config — a JSON Schema describing the shape of your test data: what fields each test case has (an input, a correct label, maybe some metadata).
2. Testing criteria — one or more graders that look at a model's output for a given test case and decide whether it's correct (Lesson 3 covers grader types in depth; this lesson uses the simplest one, string_check, so you can focus on the overall workflow first).

Once an eval exists, you don't run it directly. Instead, you create eval runs against it — each run points at a specific model, a specific prompt template, and a specific dataset file, executes the model against every row, grades every output with your testing criteria, and reports back pass/fail counts. This separation matters: the eval defines what correct means and what shape your data takes, and a run defines which model and prompt you're currently testing. That lets you run the same eval against gpt-4.1, then against a different prompt, then against a different model snapshot next month, and compare all three runs against the exact same bar.

Setup

You need the OpenAI Python SDK and an API key available as an environment variable.

Installation

pip install openai

.env

OPENAI_API_KEY=your_api_key_here

Never hard-code your API key directly in a script. Load it from the environment so it never ends up committed to source control:

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

If you're using a .env file locally, load it with python-dotenv (pip install python-dotenv) and call load_dotenv() before reading the environment variable. In production, set the environment variable through your hosting platform's secrets manager instead of shipping a .env file at all.

The Example: Categorizing Support Tickets

To keep this concrete, we'll build an eval for a realistic feature: a function that reads the text of an incoming support ticket and classifies it into one of a fixed set of categories — billing, technical, account, or other. This is exactly the kind of feature Lesson 1 warned you about: it looks trivially correct on three hand-picked examples, and it can quietly misclassify a meaningful slice of real traffic without anyone noticing until a human audits a batch of tickets by hand.

Step 1: Define the Eval

The first step is telling the API the shape of your test data and how a correct answer will be judged. Each row in your dataset will have a ticket_text field (the input) and a correct_label field (the ground truth a human assigned).

eval_config = client.evals.create(
    name="Support Ticket Categorization",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {
                "ticket_text": {"type": "string"},
                "correct_label": {"type": "string"},
            },
            "required": ["ticket_text", "correct_label"],
        },
        "include_sample_schema": True,
    },
    testing_criteria=[
        {
            "type": "string_check",
            "name": "Matches human-labeled category",
            "input": "{{ sample.output_text }}",
            "operation": "eq",
            "reference": "{{ item.correct_label }}",
        }
    ],
)

print(eval_config.id)

Walking through the important parts:

• data_source_config describes your dataset's schema using JSON Schema, the same way you'd describe a structured output shape. type: "custom" means you're bringing your own data, as opposed to referencing a stored dataset OpenAI hosts for you.
• include_sample_schema: True tells the eval that, in addition to your dataset's own fields (ticket_text, correct_label), each row will also have a sample object attached once a run actually executes — this is what lets your testing criteria reference {{ sample.output_text }}, the model's real output, rather than just the static dataset.
• testing_criteria is a list of graders. Here we use one string_check grader with the eq (exact match) operation: it passes only if the model's raw output text is character-for-character identical to the correct_label field from that row. This is strict on purpose for a first example — Lesson 3 shows softer, more forgiving grading strategies for open-ended text.
• The two template expressions, {{ sample.output_text }} and {{ item.correct_label }}, are how graders reference dynamic values. item always refers to a field from your dataset row; sample always refers to something produced by the model during the run.

client.evals.create() returns an eval object with an id — save it, because every run you create afterward needs to reference it.

Step 2: Build and Upload Your Test Dataset

Your dataset is a JSONL file — one JSON object per line — where each object matches the schema you just declared. This is where the advice from Lesson 1 about logging real production traffic pays off directly: the strongest datasets mix examples you write by hand for coverage with real (anonymized) examples pulled from logs.

tickets.jsonl

{"item": {"ticket_text": "I was charged twice for my subscription this month", "correct_label": "billing"}}
{"item": {"ticket_text": "The app crashes every time I try to upload a photo", "correct_label": "technical"}}
{"item": {"ticket_text": "I can't log in, it says my password is wrong but I just reset it", "correct_label": "account"}}
{"item": {"ticket_text": "Do you offer discounts for students?", "correct_label": "billing"}}
{"item": {"ticket_text": "Your export button doesn't work on Safari", "correct_label": "technical"}}
{"item": {"ticket_text": "I want to delete my account permanently", "correct_label": "account"}}
{"item": {"ticket_text": "What are your office hours?", "correct_label": "other"}}

A real dataset should have far more than seven rows — treat this as the minimum shape, not the target size. Aim for at least 50–100 rows for a first real eval, covering every category you care about and deliberately including the messy cases from Lesson 1: tickets that mix two topics, tickets in a casual or frustrated tone, tickets that are ambiguous even to a human reader. If two people on your team would label a ticket differently, either resolve the disagreement before adding it, or leave it out — an eval can't be more consistent than the ground truth you feed it.

Upload the file with the purpose set to "evals":

uploaded_file = client.files.create(
    file=open("tickets.jsonl", "rb"),
    purpose="evals",
)

print(uploaded_file.id)

Step 3: Create a Run

A run is where you actually specify the model and the prompt, and point the eval at your uploaded data:

run = client.evals.runs.create(
    eval_config.id,
    name="gpt-4.1 baseline",
    data_source={
        "type": "responses",
        "model": "gpt-4.1",
        "input_messages": {
            "type": "template",
            "template": [
                {
                    "role": "developer",
                    "content": (
                        "You are a support ticket classifier. Read the "
                        "ticket and respond with exactly one label: "
                        "billing, technical, account, or other. "
                        "Respond with only the label, nothing else."
                    ),
                },
                {"role": "user", "content": "{{ item.ticket_text }}"},
            ],
        },
        "source": {"type": "file_id", "id": uploaded_file.id},
    },
)

print(run.id, run.status)

Here, type: "responses" tells the run to use the Responses API to generate each output (the alternative is "completions" for the Chat Completions API). The template is a message list just like any normal Responses API call, except the user message contains {{ item.ticket_text }} — a placeholder the run engine fills in with that field from each row of your dataset before sending the request. The run executes this prompt once per row in tickets.jsonl, grades every output with the string_check criterion from Step 1, and aggregates the results.

Step 4: Retrieve and Read the Results

Runs execute asynchronously — for a large dataset, this can take a while, so you poll for status rather than blocking:

import time

while True:
    run = client.evals.runs.retrieve(run.id, eval_id=eval_config.id)
    if run.status in ("completed", "failed"):
        break
    time.sleep(5)

print(run.status)
print(run.result_counts)

result_counts gives you the headline numbers: how many rows passed, how many failed, how many errored (for example, if the model call itself threw an exception rather than returning an answer). This is the number Lesson 1 argued you can never get from manual testing — a precise, repeatable pass rate across every row in your dataset, not an impression from three anecdotes.

For a deeper breakdown, per_testing_criteria_results shows pass/fail counts broken down by each grader you defined (useful once you have more than one testing criterion), and report_url gives you a link to a dashboard view where you can inspect individual failing rows — which is usually where the real insight lives, since a single aggregate percentage tells you that something is wrong but not what.

Reading a Failure Correctly

Suppose your run comes back with 91 passed, 9 failed, out of 100. The temptation is to treat this as "9% bad, ship it anyway." Resist that until you've actually looked at which 9 rows failed. In practice, failures cluster. If all 9 failures are tickets that mention both a billing issue and a login problem in the same message, you've learned something specific and fixable: your prompt needs an instruction for how to handle ambiguous, multi-topic tickets, rather than a vague nudge to "try harder." That's a concrete, testable hypothesis you can turn into a prompt change — and then re-run the exact same eval to confirm the fix actually worked, instead of guessing.

Comparing Two Prompts With the Same Eval

The real payoff of separating the eval (what "correct" means) from the run (which model/prompt you're testing) is that you can run the exact same eval against two different prompts and get a head-to-head comparison, rather than two isolated impressions from two separate testing sessions.

Suppose the baseline run above scored 84 passed out of 100, and you suspect the model is struggling with tickets that mention a specific product name alongside a complaint. You revise the developer message to be more explicit about how to handle ambiguous or multi-topic tickets:

run_v2 = client.evals.runs.create(
    eval_config.id,
    name="gpt-4.1 with disambiguation instructions",
    data_source={
        "type": "responses",
        "model": "gpt-4.1",
        "input_messages": {
            "type": "template",
            "template": [
                {
                    "role": "developer",
                    "content": (
                        "You are a support ticket classifier. Read the "
                        "ticket and respond with exactly one label: "
                        "billing, technical, account, or other. "
                        "If a ticket mentions more than one issue, "
                        "classify it by the FIRST issue the customer "
                        "raises. Respond with only the label, nothing else."
                    ),
                },
                {"role": "user", "content": "{{ item.ticket_text }}"},
            ],
        },
        "source": {"type": "file_id", "id": uploaded_file.id},
    },
)

Because both runs point at the same eval ID and the same dataset file, their result_counts are directly comparable — same test cases, same grading logic, only the prompt changed. If run_v2 comes back at 93 passed out of 100, you have actual evidence the disambiguation instruction helped, not just a feeling that the new wording "reads better." If it comes back at 80, you've learned the new instruction made things worse in a way three manual tests would never have shown you, and you can revert with confidence instead of guesswork. This is the workflow you'll repeat constantly once evals are part of your process: change one thing, re-run, compare the number, keep or revert.

What's Actually Happening Behind the Scenes

It helps to understand the run as three distinct phases, even though the API handles all three for you automatically:

1. Templating — for every row in your dataset file, the run engine substitutes {{ item.* }} placeholders in your prompt template with that row's actual field values, producing one fully-formed prompt per row.
2. Generation — each filled-in prompt is sent to the model you specified (gpt-4.1 in the examples above), and the response is captured as that row's sample, including sample.output_text.
3. Grading — each testing criterion you defined on the eval runs against every row, comparing whatever it's configured to compare (here, sample.output_text against item.correct_label) and recording a pass or fail.

Only after all three phases finish for every row does result_counts become final, which is why a run's status moves through an in-progress state before landing on completed. Understanding this three-phase shape also makes debugging much easier: if something's wrong, the first question is which phase failed — did templating substitute the wrong value, did the model produce unexpected output, or did the grader logic have a bug? The report_url view lets you inspect all three for any individual row, which is usually faster than guessing from the aggregate numbers alone.

Real-World Use Case: Catching a Regression Before a Deploy

Consider how this fits into an actual team's workflow rather than a one-off exercise. A support tooling team keeps their ticket classifier's eval — the one built in this lesson, grown over months to 300 real, anonymized examples — wired into their continuous integration pipeline. Every pull request that touches the classifier's system prompt automatically triggers a new eval run via a small script that calls client.evals.runs.create(), polls for completion, and fails the CI check if the pass rate drops by more than two percentage points compared to the current production prompt's last recorded score.

When an engineer later tweaks the prompt to fix an unrelated complaint about verbose responses, the CI check flags a drop from 94% to 89% before the change is ever merged. Looking at the failing rows through the report URL shows the new wording accidentally made the model less confident about the other category, causing it to default to technical more often. The engineer adjusts the instruction, reruns, gets back to 95%, and merges — a regression that would have shipped silently under a "looked fine when I tested it" process is caught in minutes, with zero manual reading of transcripts. This is the practical destination this entire lesson has been building toward: an eval isn't a one-time exercise you run once and forget, it's a permanent, automatable gate your team relies on the same way it relies on any other CI check.

Common Mistakes

Schema mismatch between the eval definition and the JSONL file. If your item_schema requires correct_label but a row in your JSONL file is missing it, that row will error rather than run. Validate your JSONL against your schema locally before uploading, especially for larger datasets assembled from scripts.

Forgetting include_sample_schema: True. Without it, your testing criteria can't reference {{ sample.output_text }} at all, because the eval doesn't know a sample object will exist. This is an easy one-line miss that produces a confusing template error when you create the run.

Grading on exact string match for open-ended output. string_check with eq works well for fixed-label classification like the example above, but it will fail almost everything for a summarization or open-ended writing task, where two correct answers can be worded completely differently. That's a mismatch between the task and the grader, not a real quality problem — Lesson 3 covers softer grading approaches (text_similarity, score_model) built for exactly this case.

Treating the dataset as a one-time artifact. Datasets should grow. Every time you find a real production failure, the fix isn't just to patch the prompt — it's to also add that exact failing case to your dataset, so a future change can never silently reintroduce the same bug without the eval catching it.

Troubleshooting

Error: the run stays in in_progress far longer than expected. Large datasets and heavier models take real time to process row by row. If a run seems stuck well past what dataset size would explain, retrieve it again to confirm the exact status rather than assuming — transient API-side delays are more common than actual hangs.

Error: rows show as errored instead of failed. A failed row means the grader ran and decided the output was wrong. An errored row means something broke before grading could happen — usually a malformed template reference, a model call that hit a rate limit, or a row that doesn't match your declared schema. Check the specific error message on that row via the report URL rather than treating it the same as a normal grading failure.

Unexpectedly low pass rate on a prompt you're confident in. Double-check the exact grader operation. eq requires a character-for-character match, including case and punctuation. If your model sometimes answers "Billing" instead of "billing", that's a grader strictness problem, not a model quality problem — either tighten your prompt's output instructions or switch to the case-insensitive ilike operation covered in Lesson 3.

client.evals.runs.create() raises a validation error about the template. This usually means a {{ item.field_name }} reference in your prompt template doesn't match a field name declared in item_schema when you created the eval. The two have to agree exactly — if your dataset uses ticket_text but your template references {{ item.ticket }}, the mismatch surfaces as a template error at run-creation time rather than a silent failure later, which is a helpful early signal that something in your schema and your prompt have drifted apart.

The file upload succeeds, but every row errors out during the run. Open the JSONL file and confirm every line is valid, standalone JSON — a single misplaced comma or an unescaped quote in one line is enough to make the parser reject that row, and depending on how the file was generated, a bug in one row's formatting can sometimes repeat across every row if it came from the same buggy export script. Validating the file locally with a small script that calls json.loads() on every line before uploading catches this in seconds, well before you've spent API calls generating outputs for rows that were never going to grade correctly.

A short validation script is worth keeping around permanently rather than writing ad hoc each time:

import json

def validate_jsonl(path: str, required_fields: set[str]) -> None:
    with open(path) as f:
        for line_number, line in enumerate(f, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                row = json.loads(line)
            except json.JSONDecodeError as e:
                raise ValueError(f"Line {line_number}: invalid JSON ({e})")
            item = row.get("item", {})
            missing = required_fields - item.keys()
            if missing:
                raise ValueError(f"Line {line_number}: missing fields {missing}")

validate_jsonl("tickets.jsonl", required_fields={"ticket_text", "correct_label"})

Running this before every upload turns a class of run-time errors that used to surface only after burning API calls into an instant, local, zero-cost check.

Best Practices

Start with a small, high-quality dataset rather than a huge, noisy one — 50 carefully labeled examples you trust are worth more than 500 you're not sure about, because a noisy dataset produces a noisy, untrustworthy pass rate. Version your dataset file the same way you version code, so you can tell exactly which set of test cases a given historical run was scored against. Re-run the same eval every time you change the prompt, the model, or a tool definition, and keep a simple record — even a spreadsheet — of the pass rate over time, so a regression shows up as a visible drop rather than a mystery. And resist the urge to keep tweaking the prompt until the eval hits 100%; a dataset with a few permanently hard, ambiguous cases is often more honest than one you've quietly shaped to be easy to pass.

It's also worth deciding, upfront, who owns adding new rows to the dataset. On the team described above, the convention is simple: any time a real misclassification reaches a human — through a bug report, a support escalation, or a manual audit — that exact ticket text and its correct label get added to tickets.jsonl before the underlying prompt bug is even fixed. This means the dataset grows monotonically more representative of real failure modes over the life of the feature, instead of staying frozen at whatever a developer imagined on day one. A team that only edits the dataset when someone "gets around to it" tends to end up with an eval that looks reassuring but quietly stops reflecting reality.

Suggested Project Structure

For anything beyond a single throwaway script, it's worth splitting eval setup from eval execution, so you're not recreating the eval definition (and accidentally generating a new eval ID) every time you want to run it:

ticket-classifier-eval/
├── .env
├── requirements.txt
├── tickets.jsonl          # your test dataset
├── setup_eval.py          # creates the eval + uploads the file, run once
└── run_eval.py            # creates a run against the existing eval, run often

setup_eval.py contains the client.evals.create() and client.files.create() calls from Steps 1 and 2, and should print the resulting eval.id and file.id so you can save them (as environment variables or in a small config file) rather than hard-coding them inline everywhere. run_eval.py reads those saved IDs and only performs Steps 3 and 4 — creating a run and polling for results — which is the part you'll actually execute repeatedly, every time you touch the prompt. Keeping these concerns separate also makes it straightforward to wire run_eval.py into a CI pipeline later, since it becomes a small, self-contained script with a clear exit condition: pass if the pass rate meets your bar, fail otherwise.

Frequently Asked Questions

Do I need a separate eval for every prompt variant I want to test? No — this is precisely what runs are for. Create one eval per task (one eval for ticket classification, a separate eval for a different feature entirely), and create a new run every time you want to test a different prompt or model against that same task. Creating a brand-new eval for every prompt tweak throws away your ability to compare runs against each other, since result_counts is only meaningful when every run being compared used the same dataset and the same grading criteria.

Can I test a model I don't call through the Responses or Chat Completions API — like a fine-tuned model? Yes. The model field in a run's data_source accepts a fine-tuned model ID (the ft:... identifiers covered in Lesson 5) exactly the same way it accepts a base model name like gpt-4.1. This is, in fact, one of the most useful applications of an eval: running the same dataset and grading criteria against a base model and a fine-tuned model to get an apples-to-apples comparison of whether fine-tuning actually improved anything, instead of relying on a subjective impression.

Can I have more than one testing criterion on the same eval? Yes — testing_criteria accepts a list, and a run reports pass/fail separately for each one via per_testing_criteria_results, while result_counts reflects the combined outcome. This is useful even before you get to the multi grader covered in Lesson 3: for the ticket classifier, you might add a second, independent criterion checking that the output contains no extra text beyond the label itself, so a technically-correct-but-verbose answer ("billing, I think") doesn't get silently counted the same as a clean one.

What happens if I need to update the dataset after the eval already has runs against it? Upload a new file with client.files.create() and reference the new file's ID in your next run's source. Previous runs stay tied to the file they were originally run against, so your historical results remain valid and comparable to each other — only runs going forward will be graded against the updated dataset. This is also a natural point to give your dataset file a version marker in its filename (tickets_v2.jsonl) so it's obvious from the file list alone which version any given run used.

Lesson 3 goes deep on the grader types only briefly introduced here — how to score open-ended text, how to use a second model as a judge of subjective quality, and how to write fully custom Python grading logic for cases no built-in grader handles well.