Status Note

A Quick Status Note

Graders, as documented here, are part of the same Evals platform covered in Lesson 2, which OpenAI has scheduled to become read-only on October 31, 2026 and shut down on November 30, 2026. That timeline doesn't change anything about this lesson's content: the five grading strategies below — exact matching, similarity scoring, LLM-as-judge, custom code, and weighted combinations — are the standard vocabulary for scoring model output across essentially every evaluation tool on the market, including the third-party options OpenAI points to as a migration path. Learn the concepts here; they'll transfer directly to whatever tool you're running evals with by the time you read this.

What a Grader Actually Is

In Lesson 2, the testing_criteria field on your eval used a single string_check grader to decide whether the model's output exactly matched a human-labeled category. That was the simplest possible grader on purpose, so the overall eval workflow — dataset, run, results — could stay the focus. This lesson goes deep on the thing that actually decides pass or fail: the grader itself.

A grader is a small, well-defined function — sometimes literal Python code, sometimes a call to another model, sometimes a string comparison — that takes two things, the model's output (sample) and the dataset row it came from (item), and produces a score. For most grader types that score is a number between 0 and 1, and an optional pass_threshold decides how high that number needs to be to count as a pass. Choosing the right grader for a task is arguably the single most important design decision in building a useful eval, because a badly chosen grader can make a genuinely good model look broken, or — more dangerously — make a genuinely broken model look fine.

OpenAI's evaluation guidance is explicit about this: prefer graders that produce gradual scoring over strict binary pass/fail wherever the task allows it, because a 0-or-1 grader throws away useful information about how wrong an answer was, and can hide slow, gradual degradation that would otherwise show up as a shrinking average score long before it becomes an outright failure.

There are five grader types available: string_check, text_similarity, score_model, python, and multi. This lesson works through each one with a real use case, because picking between them is a practical skill you build by seeing where each one fits — and where it doesn't.

Grader 1: String Check — Exact and Substring Matching

You already saw this one in Lesson 2. Its full shape:

{
  "type": "string_check",
  "name": "Matches human-labeled category",
  "input": "{{ sample.output_text }}",
  "operation": "eq",
  "reference": "{{ item.correct_label }}"
}

Four operations are available:

When to use it: Fixed-vocabulary outputs where there's exactly one correct string — classification labels, yes/no answers, a specific extracted value like a status code. It is the cheapest grader to run (no extra model call, no code execution) and the easiest to reason about, which makes it the right default whenever the task genuinely has one correct answer.

When it breaks down: Anything open-ended. If your task is "summarize this article" or "write a helpful response to this customer," there is no single correct string to match against — two summaries can both be excellent and share almost no words in common. Forcing string_check onto an open-ended task produces a misleadingly low pass rate that reflects wording differences, not actual quality differences. This is one of the most common mistakes teams make when they build their first eval: reusing the grader from a classification task on a generation task, because it's already there and it's simple.

Grader 2: Text Similarity — Scoring Close-Enough Answers

For tasks where there's a reference answer but exact wording shouldn't matter, text_similarity scores how close the model's output is to a reference using a choice of established similarity metrics:

{
  "type": "text_similarity",
  "name": "Close to reference answer",
  "input": "{{ sample.output_text }}",
  "reference": "{{ item.reference_answer }}",
  "pass_threshold": 0.75,
  "evaluation_metric": "fuzzy_match"
}

The evaluation_metric field accepts several options, each suited to a different kind of closeness:

• fuzzy_match — edit-distance-based similarity, good for catching near-identical strings with small typos or formatting differences.
• bleu and gleu — n-gram overlap metrics originally built for machine translation quality; useful when word choice and order should roughly match a reference.
• meteor — similar to BLEU but more tolerant of synonyms and word-order variation.
• cosine — compares the two texts as embeddings and measures the angle between them, which captures semantic similarity even when the wording is completely different.
• rouge_1 through rouge_l — overlap metrics commonly used for summarization quality, comparing shared words or shared subsequences between the candidate and reference.

When to use it: Paraphrase-tolerant tasks where you have a solid reference answer but expect legitimate wording variation — translation, summarization, or short-answer extraction where "12 units" and "twelve units" should both count as correct. cosine in particular is worth reaching for whenever the meaning matters more than the phrasing.

When it breaks down: Tasks where there genuinely isn't a single reference answer to compare against — open-ended creative writing, or subjective quality judgments like "was this response empathetic." No similarity-to-reference metric can capture that, because there's no one reference text that defines "empathetic." That's what the next grader is for.

Grader 3: Score Model — Using an LLM as the Judge

For subjective or open-ended quality — tone, helpfulness, adherence to a style guide, whether a response appropriately declined an out-of-scope request — you need a grader that can actually understand the response, not just compare it to a fixed string. score_model does this by sending the sample to a separate model call, along with grading instructions you write, and asking that model to return a numeric judgment.

{
  "type": "score_model",
  "name": "Response quality judge",
  "model": "gpt-4.1",
  "input": [
    {
      "role": "developer",
      "content": "You are grading a customer support response for helpfulness and tone. Score from 0 to 1, where 1 means the response fully resolves the customer's issue in a warm, professional tone, and 0 means it is unhelpful, rude, or off-topic. Return only the numeric score."
    },
    {
      "role": "user",
      "content": "Customer message: {{ item.ticket_text }}\n\nSupport response: {{ sample.output_text }}"
    }
  ],
  "pass_threshold": 0.7,
  "range": [0, 1],
  "sampling_params": {
    "temperature": 0,
    "reasoning_effort": "medium"
  }
}

The grader model can be one of several supported judge models — gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18, the gpt-4.1 family, or the o1/o3/o4-mini reasoning model family — and sampling_params lets you control things like temperature (set it to 0 for judging, since you want the grader itself to be as consistent as possible, not creative) and reasoning_effort for reasoning-capable judge models. The grader returns a structured result containing both the numeric result and, usefully, its reasoning for that score, which is invaluable when you're debugging why a particular row failed — you can read why the judge scored it low, not just that it did.

When to use it: Any dimension of quality that requires actual comprehension and judgment rather than mechanical comparison — tone, safety, instruction-following, whether a response stayed within a specified persona, whether an answer is factually grounded in a provided document. This is the grader you reach for once your task moves past "is this the right label" into "is this a good response."

The single most important best practice for this grader is writing a precise, unambiguous scoring rubric in the developer message, the same way you'd write instructions for a new human contractor grading the same task. A vague instruction like "score how good this is" produces an inconsistent, noisy judge — the model will make up its own criteria, and those criteria can drift between calls. A specific rubric, ideally with concrete examples of what a 0, a 0.5, and a 1 look like, produces a judge that grades consistently enough to trust across hundreds of rows and across multiple runs over time.

A real risk to watch for: reward hacking. If you use a score_model grader as a training signal (as in reinforcement fine-tuning) or lean on it heavily to guide prompt iteration, it's possible for a prompt to learn to satisfy the letter of the judge's rubric without actually being good — for example, a support response that stuffs in empathetic-sounding phrases ("I completely understand how frustrating this must be!") without actually solving the customer's problem, if your rubric only checks for warmth and not resolution. Guard against this by writing rubrics that check for the actual outcome you care about, not just surface signals correlated with it, and by periodically spot-checking high-scoring rows by hand to confirm the judge and your own judgment still agree.

Grader 4: Python — Custom Logic for Everything Else

Sometimes correctness is a computation, not a comparison or a judgment call — validating that output is parseable JSON matching a schema, checking that a generated SQL query only touches allow-listed tables, confirming a numeric answer falls within a tolerance of the true value. For cases like this, the python grader lets you write the exact scoring logic yourself:

{
  "type": "python",
  "name": "Valid JSON with required fields",
  "source": "def grade(sample, item):\n    import json\n    try:\n        parsed = json.loads(sample['output_text'])\n    except json.JSONDecodeError:\n        return 0.0\n    required = {'order_number', 'issue_type'}\n    if not required.issubset(parsed.keys()):\n        return 0.0\n    return 1.0"
}

Your grade function receives two plain dictionaries:

• sample — contains the model's output in several forms: output_text (the raw text), output_json (if the model was asked for structured output), output_tools (any tool/function calls the model made), and output_audio if relevant.
• item — the fields from your dataset row, exactly as you defined them in the eval's item_schema.

The function returns a numeric score, which the run engine records exactly like any other grader's result. This gives you the full power of Python — parsing, validation, calling other libraries — as your correctness check.

There are real constraints to know about before you reach for this grader on anything heavy: code is capped at 256KB, execution is limited to 2 minutes, the sandbox provides 2GB of memory, 1GB of disk, and 2 CPU cores, and critically, there is no network access — you cannot call another API, another model endpoint, or an external database from inside a python grader. The environment does come with a solid set of scientific and text-processing packages preinstalled, including numpy, scipy, pandas, scikit-learn, rapidfuzz, and NLTK, so most validation and scoring logic that doesn't need network access is straightforward to write.

When to use it: Structural correctness (valid JSON, valid schema, valid SQL syntax), numeric tolerance checks, anything involving regular expressions or parsing that's awkward to express as a string match, or combining several mechanical checks into one score. It's the right tool whenever you find yourself thinking "I could write this check in five lines of Python" — because you can, directly, instead of trying to force the logic into a string-matching grader it doesn't fit.

When it breaks down: Anything requiring judgment about meaning or quality — a python grader has no understanding of language, it only executes the logic you give it. If the correctness criterion is "does this response sound professional," no amount of Python will capture that; that's score_model territory.

Grader 5: Multi — Combining Several Graders Into One Score

Real tasks often have more than one thing that needs to be correct simultaneously. A tool-calling agent needs to call the right tool and pass the right arguments — getting one right and one wrong shouldn't necessarily count as a full pass or a full failure. The multi grader runs several named sub-graders and combines their individual scores using a formula you define:

{
  "type": "multi",
  "name": "Correct tool call",
  "graders": {
    "correct_function": {
      "type": "string_check",
      "input": "{{ sample.output_tools[0].function.name }}",
      "reference": "{{ item.expected_function }}",
      "operation": "eq"
    },
    "correct_arguments": {
      "type": "text_similarity",
      "input": "{{ sample.output_tools[0].function.arguments }}",
      "reference": "{{ item.expected_arguments }}",
      "evaluation_metric": "fuzzy_match"
    }
  },
  "calculate_output": "0.5 * correct_function + 0.5 * correct_arguments"
}

Each key inside graders is a sub-grader with its own type and configuration — you can mix and match any of the four grader types covered above, including nesting a score_model judge alongside a string_check. The calculate_output field is a formula string that combines the named sub-scores; it supports the standard arithmetic operators (+, -, *, /, ^) and functions like min, max, abs, floor, ceil, exp, sqrt, and log, so you can weight some checks more heavily than others, or require the minimum of two scores rather than their average if both genuinely need to be right for the output to count.

When to use it: Any task where correctness has multiple independent dimensions — a tool call needing both the right function and the right arguments, a structured extraction task needing several fields all present and correct, or a response that needs to be both factually accurate (checked with one grader) and appropriately toned (checked with a score_model grader). Combining these into a single weighted score gives you one clean number to track over time while still preserving the ability to look at per_testing_criteria_results and see exactly which sub-dimension is dragging the score down.

The Templating System, in Full

Every grader type uses the same {{ namespace.field }} templating syntax, and it's worth being precise about what each namespace resolves to, since a subtle mismatch here is one of the most common sources of grader misconfiguration:

• item.* always refers to a field on the current dataset row, exactly as declared in your eval's item_schema. If your schema doesn't declare a field, referencing it in a template produces an error, not an empty string.
• sample.* always refers to something produced during the run: sample.output_text for plain text, sample.output_json for structured outputs, sample.output_tools for tool/function calls (an array, so a specific call is addressed as sample.output_tools[0]), and sample.output_audio for audio outputs where relevant.

Both namespaces are available in every grader type's input and reference fields, which is what makes it possible to build a multi grader that mixes a mechanical string_check against item.expected_function with a subjective score_model judgment about sample.output_text, all within the same testing criterion.

Choosing the Right Grader: A Practical Checklist

When you're not sure which grader fits a new eval, work through these questions in order:

1. Is there exactly one correct string, character for character? Use string_check.
2. Is there a reference answer, but wording can legitimately vary? Use text_similarity, and pick cosine if meaning matters more than word choice, or one of the n-gram metrics (bleu, rouge_l) if structural overlap matters.
3. Does correctness require actual judgment — tone, helpfulness, safety, subjective quality? Use score_model, and invest real effort in writing a precise rubric.
4. Is correctness a computation or a structural check — valid JSON, a numeric tolerance, a regex match? Use python.
5. Does the task have more than one independent thing that needs to be right at once? Use multi to combine the graders above into one weighted score.

Real-World Example: Grading a Document Q&A Assistant

To see several grader types work together, consider a Q&A assistant that answers questions using an internal knowledge base and cites which document it pulled the answer from. There are three genuinely different things that need to be correct here, and forcing them all into one grading approach would hide exactly which one is failing when the overall score drops.

{
  "type": "multi",
  "name": "Document QA correctness",
  "graders": {
    "cites_correct_document": {
      "type": "string_check",
      "input": "{{ sample.output_json.source_document_id }}",
      "reference": "{{ item.correct_document_id }}",
      "operation": "eq"
    },
    "answer_is_accurate": {
      "type": "score_model",
      "model": "gpt-4.1",
      "input": [
        {
          "role": "developer",
          "content": "Score 1 if the answer is factually consistent with the reference answer and does not contradict or invent information beyond it. Score 0 if it contradicts the reference or adds unsupported claims. Score 0.5 for a partially correct or incomplete answer."
        },
        {
          "role": "user",
          "content": "Question: {{ item.question }}\nReference answer: {{ item.reference_answer }}\nModel answer: {{ sample.output_json.answer }}"
        }
      ],
      "pass_threshold": 0.5,
      "range": [0, 1],
      "sampling_params": {"temperature": 0}
    },
    "valid_response_shape": {
      "type": "python",
      "source": "def grade(sample, item):\n    required = {'answer', 'source_document_id'}\n    return 1.0 if required.issubset(sample.get('output_json', {}).keys()) else 0.0"
    }
  },
  "calculate_output": "0.2 * valid_response_shape + 0.3 * cites_correct_document + 0.5 * answer_is_accurate"
}

This single testing criterion checks three independent failure modes at once: a python grader confirming the response even has the right shape before anything else is checked (worth relatively little on its own, since it's a baseline sanity check rather than a measure of quality), a string_check confirming the citation points at the right source document, and a score_model judge weighing most heavily, since factual accuracy is the dimension that actually matters most to a user. If the combined score drops in a future run, per_testing_criteria_results combined with the individual sub-scores tells you immediately whether the regression is a citation problem, an accuracy problem, or a response-format problem — three very different bugs that would require three completely different fixes, and that a single opaque pass/fail number would have flattened into one undifferentiated "it got worse."

Cost and Speed Tradeoffs Between Grader Types

Grader choice isn't only a matter of which one measures the right thing — it also affects how expensive and how slow your eval runs are, which matters once your dataset grows past a handful of rows or you start running evals on every pull request.

string_check and text_similarity are essentially free and near-instant: they're pure computation on text you already have, with no extra network calls. python graders run in a sandboxed environment and have real but small overhead — well within the 2-minute limit for anything reasonable, but not instantaneous, especially if your grading logic does nontrivial computation like parsing large documents. score_model is the most expensive and slowest of the five, because every single row triggers an entirely separate model API call, on top of the model call that generated the original sample being graded — for a 500-row dataset, that's 500 additional inference calls, each with its own latency and token cost, every time you run the eval.

This has a practical consequence for how you design evals with many rows: reserve score_model for the specific dimensions of quality that genuinely require judgment, and use the cheaper mechanical graders for everything that can be checked mechanically. The document Q&A example above follows this principle deliberately — the response-shape check and the citation check use fast, free graders, and only the genuinely subjective "is this answer accurate" dimension pays the cost of an LLM judge. An eval built entirely out of score_model graders, when half of them could have been a string_check or a python check instead, is both slower to run and more expensive than it needs to be, for no gain in the quality of what you're measuring.

Common Mistakes

Defaulting to binary pass/fail everywhere. OpenAI's own guidance is to prefer gradual, 0-to-1 scoring wherever the task allows it, because a binary grader hides the difference between "barely failed" and "catastrophically wrong," and hides slow degradation until it crosses the pass/fail line all at once.

Writing a vague rubric for a score_model grader. "Rate how good this response is" invites the judge model to invent its own inconsistent criteria. Always specify concretely what a 0, a mid-range score, and a 1 look like, ideally with a short example of each.

Letting a score_model grader reward surface signals instead of real outcomes. A rubric that only checks tone without checking whether the underlying problem was actually solved is exploitable — a model (or a well-meaning prompt engineer chasing the number) can learn to sound good without being good. Write rubrics around the actual outcome you care about.

Using an unbalanced dataset with a skewed label distribution. If 95% of your dataset rows have the same correct label, a model that always guesses that label scores 95% while being useless at the actual task. Balance your dataset deliberately, or track per-category pass rates instead of relying on the single overall number.

Forgetting the python grader has no network access. If your grading logic needs to call another API — an embeddings endpoint, a database lookup — it cannot run inside a python grader. You'd need to precompute whatever that step provides and include it as a field in your dataset row instead.

Troubleshooting

A score_model grader gives inconsistent scores across otherwise-identical reruns. Confirm sampling_params.temperature is set to 0. A judge model sampling with a nonzero temperature will vary its own scoring even when nothing about the sample being graded has changed, which defeats the purpose of a repeatable eval.

A python grader run fails immediately. Check the code against the constraints: total source under 256KB, no import of a package outside the provided scientific/text-processing set, and no attempted network calls (requests, urllib, or similar will fail in the sandboxed environment). Also confirm your function is literally named grade and accepts exactly the sample and item parameters — a different function signature won't be picked up.

A multi grader's calculate_output throws a formula error. The formula string references sub-grader names exactly as declared as keys in the graders object — a typo in that name, or referencing a key that doesn't exist, breaks the formula. Double-check the key names match precisely, including case.

Scores all cluster suspiciously near the pass_threshold. This can indicate the grading criteria are ambiguous enough that the judge (or your own threshold choice) is essentially guessing. Consider whether the task needs a clearer rubric, a different metric, or whether the underlying task itself is genuinely ambiguous and needs a clearer specification before grading it makes sense at all.

Best Practices

Favor gradual scoring over binary pass/fail whenever the task supports it, since a 0-to-1 score preserves information a binary grader throws away. Write score_model rubrics the way you'd write instructions for a new hire — specific, with worked examples of each score level — rather than a one-line vibe description. Keep an eye on reward hacking: periodically read a sample of high-scoring rows yourself and confirm you'd actually agree they're good, not just that they satisfied the letter of the rubric. Balance your datasets so a lazy, one-size-fits-all output can't score artificially well. And when a task has multiple independent correctness dimensions, prefer a multi grader with clear per-dimension results over cramming everything into a single opaque score — you want to know not just that something failed, but which part of it failed, so you know what to actually fix.

A Note on Grading Multi-Turn and Agentic Transcripts

Everything above has used single-turn examples for clarity, but every grader type applies just as directly when item and sample represent an entire conversation or an agent's full tool-calling trajectory rather than one exchange. A python grader can walk a list of tool calls in sample.output_tools and check that a required lookup happened before a final answer was given, not just that the final answer looks right. A score_model judge can be handed the full transcript and asked to rate whether the agent recovered gracefully after a failed tool call, rather than just rating the last message in isolation. The one practical adjustment worth making for longer transcripts is keeping your score_model rubric explicit about which parts of the transcript matter for the score — a judge given a ten-turn conversation and a vague rubric will tend to anchor on the most recent messages and under-weight something that went wrong three turns earlier, the same way a human skimming quickly might.

Graders live inside the Evals workflow this unit has been teaching, but the same scoring concepts — mechanical checks, similarity metrics, LLM-as-judge, and custom code — reappear anywhere models need to be evaluated, including in reinforcement fine-tuning, where a grader's score becomes a training signal rather than just a report. With eval design and grading covered, the next two lessons shift from measuring quality to changing model behavior directly: Lesson 4 covers how to decide whether fine-tuning is actually the right tool for a given problem, and Lesson 5 walks through a complete supervised fine-tuning job from data preparation to a working custom model.