How to Evaluate Prompt Quality
Why Prompts Need Evaluation A prompt that works on the three examples you tried is not necessarily a good prompt. Throughout this tutorial, many lessons have said "test it on a set of examples". This section of the tutorial turns that advice into a practice: prompt evaluation, the systematic measurement of how well a prompt performs.
Evaluation answers questions you cannot answer by reading a few outputs:
How often does the prompt produce the correct result? Which kinds of input does it fail on? Is version 2 actually better than version 1, or just different? Did a model update change its behavior? Is it good enough to put in front of users? Lesson Topic 114 Evaluation framework: what to measure and how (this lesson) 115 Creating test cases 116 Comparing outputs with a rubric 117 Evaluating accuracy, relevance, and completeness 118 Regression testing for prompt changes 119 Building an evaluation dataset 120 Common causes of inconsistent outputs What "Quality" Means for a Prompt Prompt quality is not a single number. It has several dimensions, and which ones matter depends on the task.
Dimension Question Example Failure Correctness Is the output right? Wrong label, wrong amount Faithfulness Is every claim supported by the input or sources? Invented refund timeline (Lesson 106) Format compliance Does the output follow the required structure? Text before the JSON Completeness Does it cover everything required? Answer ignores the second question Relevance Does it stay on what was asked? Long unrelated explanation Tone and style Does it suit the audience? Harsh wording to a confused student Safety and policy Does it follow rules and avoid harm? Promises a refund the policy does not allow Consistency Does it behave the same way on similar inputs and repeated runs? Same ticket gets different labels Cost and latency Is it efficient enough? 3,000-token prompt for a one-word label Choosing Dimensions for Your Task Task Most Important Dimensions Classification Correctness, format, consistency Information extraction Correctness per field, format, faithfulness RAG question answering Faithfulness, correctness, completeness Customer replies Correctness, tone, policy compliance Summarization Faithfulness, completeness, length Select a few dimensions that matter most, rather than measuring everything. As in Lesson 66, choose a primary metric and treat the others as constraints.
Evaluation Methods Each dimension can be measured in different ways, which vary in cost, speed, and reliability.
Method How It Works Best For Limitations Exact match Output equals the expected value Labels, short answers, extracted values Fails for correct answers worded differently Rule-based checks Code checks properties, such as valid JSON, length, required words, forbidden words Format, length, simple policy rules Cannot judge meaning Reference comparison Compare with a reference answer, by overlap or similarity Summaries, translations Similar wording is not the same as correct meaning Model judge A separate prompt scores the output against criteria Tone, completeness, faithfulness Must itself be validated; can be biased (Lesson 69) Human review People rate or compare outputs Final quality, subtle judgment, high-stakes cases Slow and costly; needs clear guidelines Use the Cheapest Reliable Method Prefer exact match and rule-based checks wherever they are reliable: they are free, instant, and consistent. Use model judges for dimensions code cannot measure, and people for calibration, difficult cases, and final decisions.
A Simple Evaluation Harness An evaluation harness is code that runs checks on a set of outputs and summarizes the results. The example below evaluates a support prompt that must return JSON with a label and a reply. It measures three dimensions with code: format compliance, label correctness, and a rule-based reply check.
import json import re from statistics import mean
CASES = [ {"id": "c1", "input": "Refund for Django course?", "expected_label": "Billing", "output": '{"label": "Billing", "reply": "Refunds are available within 7 days of purchase."}', "seconds": 1.2}, {"id": "c2", "input": "Can't log in", "expected_label": "Access", "output": '{"label": "Technical", "reply": "Please reset your password from the login page."}', "seconds": 0.9}, {"id": "c3", "input": "Video keeps buffering", "expected_label": "Technical", "output": 'Sure! {"label": "Technical"}', "seconds": 1.5}, {"id": "c4", "input": "Invoice please", "expected_label": "Billing", "output": '{"label": "Billing", "reply": "We will send your invoice in 5 minutes guaranteed!!!"}', "seconds": 1.1}, ]
METRICS = ("format_ok", "label_correct", "reply_ok")
def check_case(case: dict) -> dict: result = {"id": case["id"], "format_ok": False, "label_correct": False, "reply_ok": False} try: data = json.loads(case["output"]) except json.JSONDecodeError: return result result["format_ok"] = set(data) == {"label", "reply"} result["label_correct"] = data.get("label") == case["expected_label"] reply = data.get("reply", "") result["reply_ok"] = bool(reply) and len(reply) <= 200 and not re.search(r"guarantee|!!", reply, re.IGNORECASE) return result
def summarize(cases: list[dict]) -> tuple[dict, dict]: results = [check_case(case) for case in cases] summary = {metric: round(mean(r[metric] for r in results), 2) for metric in METRICS} summary["avg_seconds"] = round(mean(case["seconds"] for case in cases), 2) failures = {r["id"]: [m for m in METRICS if not r[m]] for r in results} return summary, {case_id: failed for case_id, failed in failures.items() if failed}
summary, failures = summarize(CASES) print(summary) print(failures) Output:
{'format_ok': 0.75, 'label_correct': 0.5, 'reply_ok': 0.5, 'avg_seconds': 1.18} {'c2': ['label_correct'], 'c3': ['format_ok', 'label_correct', 'reply_ok'], 'c4': ['reply_ok']} Code Explanation CASES Each case contains the input, the expected label, the model's output, and the response time. In a real harness, outputs come from running the prompt; here they are fixed so the evaluation logic can be tested on its own.
check_case()
Format: the output must be valid JSON with exactly the keys label and reply. Correctness: the label must equal the expected label (exact match). Reply rule: the reply must exist, be at most 200 characters, and must not contain "guarantee" or "!!", which represent forbidden promises and unprofessional tone. Real rule sets are longer and specific to your policies. summarize()
Averages each metric across cases. Python treats True as 1 and False as 0, so the average is the pass rate. Lists which checks each case failed, so problems can be investigated individually. Reading the Results c2 has a wrong label: a login problem labeled Technical instead of Access. This is a boundary problem between categories (Lesson 18). c3 fails all checks because of "Sure!" before the JSON. Its label was actually correct, but the format failure hides it: code cannot use an output it cannot parse. Format reliability is often the first thing to fix. c4 passes format and correctness but fails the reply rule because of an unsupported promise. Without the per-case failure list, a 50% label accuracy would not tell you that one of the "wrong" labels was a format problem. Always look at individual failures, not only the summary.
The Evaluation Workflow
- Define the task and the dimensions that matter (primary metric + constraints)
- Build test cases with expected results or criteria (Lessons 115, 119)
- Run the prompt on every test case and store the outputs
- Score each output: code checks first, then model judges, then human review where needed
- Summarize metrics and list failures
- Inspect failures, group them by cause
- Improve the prompt (Lesson 12), then run the same evaluation again
- Keep the test set for regression testing (Lesson 118) Store Outputs, Not Just Scores Save every output with its prompt version and model (Lesson 67). Stored outputs let you re-score them later with improved checks, compare versions side by side, and show concrete examples when discussing changes.
How Much Evaluation Is Enough? Situation Reasonable Starting Point Early prototype 20–30 varied cases, read manually Before releasing a feature 100+ cases covering typical, edge, and adversarial inputs Comparing two prompt versions Enough cases for differences to exceed uncertainty (Lesson 66) High-stakes decisions Larger sets, human review, and ongoing monitoring These numbers are starting points, not rules. The test set should grow over time, especially with real failures found in production.
Offline and Online Evaluation Type What It Measures Example Offline Performance on a fixed test set before release Label accuracy on 150 tickets Online Behavior with real users after release Rate of handovers, user ratings, complaints Both are needed. Offline evaluation catches problems before users see them. Online evaluation reveals inputs and situations your test set did not include (Lesson 69). Add the failures you find online to the offline test set.
Common Mistakes Mistake 1: Judging by a Few Examples What it is: Concluding that a prompt works after trying it a few times.
Why it happens: The examples looked good.
What it causes: Failures on inputs you did not try, discovered by users.
How to fix it: Evaluate on a varied test set before relying on a prompt.
Mistake 2: One Overall Score What it is: Reducing quality to a single "good / bad" or average score.
Why it happens: One number is easy to report.
What it causes: Problems in specific dimensions, such as format or tone, stay hidden.
How to fix it: Measure each important dimension separately and list per-case failures.
Mistake 3: Using Expensive Methods for Everything What it is: Using a model judge or human review for checks code could do.
Why it happens: Model judges are flexible and easy to set up.
What it causes: Higher cost, slower evaluation, and inconsistent results.
How to fix it: Use code checks first; reserve judges and people for what code cannot measure.
Mistake 4: Not Inspecting Failures What it is: Looking only at summary percentages.
Why it happens: Summaries seem sufficient.
What it causes: Wrong conclusions, such as treating a format failure as a reasoning failure.
How to fix it: Read failing cases and group them by cause.
Mistake 5: Changing the Test Set Between Comparisons What it is: Evaluating version 1 on one set and version 2 on another.
Why it happens: New cases are added while working.
What it causes: Differences in scores reflect different test cases, not different prompts.
How to fix it: Compare versions on the same test set; add new cases between comparison rounds.
Best Practices Define quality dimensions for each task. Choose a primary metric and constraints.
Use the cheapest reliable method per dimension. Code checks, then model judges, then people.
Measure dimensions separately. And always list per-case failures.
Store outputs with versions. Re-score and compare later.
Evaluate offline before release and online after. Feed online failures back into the test set.
Evaluate continuously. Prompts, models, and inputs change over time.
Try It Yourself Exercise 1: Choose Dimensions For three prompts you use, such as a summarizer, a classifier, and a reply writer, list the three most important quality dimensions for each and the evaluation method you would use for each dimension.
Exercise 2: Build a Harness Adapt check_case() and summarize() for one of your prompts. Run the prompt on 20 inputs, store the outputs, and produce a summary and a failure list. Group the failures by cause.
Exercise 3: Format vs Content Find cases where your prompt's output fails format checks. Extract the intended answer manually from each, and check whether the content was correct. Calculate how much of your apparent "accuracy problem" is really a format problem.