Comparing Prompt Outputs with a Rubric

What Is a Rubric? A rubric is a scoring guide that defines the criteria for judging an output and describes what each score level looks like for each criterion. Teachers use rubrics to grade essays consistently; prompt engineers use them to evaluate outputs that have no single correct answer, such as replies, summaries, and explanations.

Without a rubric, "Is this reply good?" depends on who is judging, their mood, and what they happen to notice. With a rubric, the question becomes several specific questions with defined answers:

Criterion 1 2 3 Accuracy Contains a factual error or unsupported promise All facts correct, but one is vague or imprecise All facts correct and precise Completeness Misses the main question Answers the main question, misses a secondary part Answers every part of the question Tone Rude, cold, or overly casual Polite but stiff or generic Warm, clear, and professional Conciseness Much longer than needed or padded Slightly long Says what is needed, no more The descriptors in each cell are what make a rubric useful. "Accuracy: 2" means nothing on its own; "All facts correct, but one is vague" tells every judge, human or model, exactly what to look for.

Lesson 114 introduced evaluation dimensions and methods. This lesson shows how to design rubrics, use them with model judges, combine scores, compare the outputs of different prompt versions, and check that the judge agrees with people.

Designing a Rubric Step 1: Choose Criteria From the Task Select the dimensions that matter for this task (Lesson 114). For a support reply: accuracy, completeness, tone, conciseness. For a summary: faithfulness, coverage, length. Four or five criteria are usually enough; more criteria make scoring slower and less consistent.

Step 2: Define Few Levels With Clear Descriptors Three levels are often easier to apply consistently than five or ten. With many levels, judges struggle to tell the difference between, for example, a 6 and a 7.

Good Descriptor Weak Descriptor "All facts correct, but one is vague or imprecise" "Mostly accurate" "Answers the main question, misses a secondary part" "Fairly complete" "Contains a factual error or unsupported promise" "Inaccurate" Good descriptors describe observable features of the output.

Step 3: Assign Weights Some criteria matter more than others. For a support reply, accuracy usually matters far more than conciseness:

Criterion Weight Accuracy 0.4 Completeness 0.3 Tone 0.2 Conciseness 0.1 Step 4: Mark Critical Criteria Weights alone can hide serious failures. An answer that is completely wrong but polite, complete-sounding, and concise could still score fairly high. Mark criteria where a low score means the output fails regardless of other scores. Accuracy is usually such a critical criterion.

Using a Rubric With a Model Judge The Judge Prompt You are evaluating a support reply using the rubric below.

accuracy: 1 = contains a factual error or a promise not supported by the policy 2 = all facts correct, but one is vague or imprecise 3 = all facts correct and precise completeness: 1 = misses the main question 2 = answers the main question, misses a secondary part 3 = answers every part of the question tone: 1 = rude, cold, or overly casual 2 = polite but stiff or generic 3 = warm, clear, and professional conciseness: 1 = much longer than needed or padded 2 = slightly long 3 = says what is needed, no more

Use the policy in as the source of truth for accuracy.

For each criterion, first quote the part of the reply that justifies your score in "evidence", then give the score. Judge each criterion independently.

Return only JSON: {"accuracy": {"evidence": "...", "score": 1-3}, "completeness": {"evidence": "...", "score": 1-3}, "tone": {"evidence": "...", "score": 1-3}, "conciseness": {"evidence": "...", "score": 1-3}}

{policy} {question} {reply} Why Each Part Matters Part Purpose Full descriptors for every level The judge applies your definitions, not its own Policy as the source of truth Accuracy is judged against facts, not the judge's general knowledge Evidence before score The judge points to the text before deciding, similar to reasoning before answering (Lesson 39); the evidence also makes scores reviewable "Judge each criterion independently" Reduces the halo effect, where a good impression on one criterion raises the others JSON output Scores can be validated and combined in code (Lesson 99) Combining Scores in Code The judge produces scores per criterion. Combining them, applying weights, and enforcing critical criteria should be done in code, not by the judge, so the rules are exact and consistent.

RUBRIC = { "accuracy": {"weight": 0.4, "levels": 3}, "completeness": {"weight": 0.3, "levels": 3}, "tone": {"weight": 0.2, "levels": 3}, "conciseness": {"weight": 0.1, "levels": 3}, } CRITICAL = {"accuracy": 2} # minimum acceptable score for critical criteria

def weighted_score(scores: dict[str, int], rubric: dict = RUBRIC) -> float: """Combine 1-to-N scores into a 0-100 weighted score, after validating them.""" problems = [f"{c}: missing" for c in rubric if c not in scores] problems += [f"{c}: {s} outside 1-{rubric[c]['levels']}" for c, s in scores.items() if c in rubric and not 1 <= s <= rubric[c]["levels"]] if problems: raise ValueError("; ".join(problems)) total = sum(rubric[c]["weight"] * (scores[c] - 1) / (rubric[c]["levels"] - 1) for c in rubric) return round(100 * total, 1)

def verdict(scores: dict[str, int], pass_mark: float = 70) -> str: for criterion, minimum in CRITICAL.items(): if scores[criterion] < minimum: return f"fail ({criterion} below {minimum})" return "pass" if weighted_score(scores) >= pass_mark else "fail (score below pass mark)"

accurate_but_incomplete = {"accuracy": 3, "completeness": 2, "tone": 3, "conciseness": 1} polite_but_wrong = {"accuracy": 1, "completeness": 3, "tone": 3, "conciseness": 3} average_everywhere = {"accuracy": 2, "completeness": 2, "tone": 2, "conciseness": 2}

print(weighted_score(accurate_but_incomplete), verdict(accurate_but_incomplete)) print(weighted_score(polite_but_wrong), verdict(polite_but_wrong)) print(weighted_score(average_everywhere), verdict(average_everywhere))

try: weighted_score({"accuracy": 4, "tone": 2, "conciseness": 2}) except ValueError as error: print("Rejected:", error) Output:

75.0 pass 60.0 fail (accuracy below 2) 50.0 fail (score below pass mark) Rejected: completeness: missing; accuracy: 4 outside 1-3 Code Explanation weighted_score()

Validation first: every criterion must be present, and every score must be within its range. The judge's output is never trusted blindly; the last example is rejected because completeness is missing and accuracy is 4 on a 1–3 scale. Normalization: (score - 1) / (levels - 1) converts a score from 1–3 into 0–1, so a 1 becomes 0, a 2 becomes 0.5, and a 3 becomes 1. Weighting: each normalized score is multiplied by its weight, the results are added, and the total is shown on a 0–100 scale. verdict()

Checks critical criteria before the weighted score. The "polite but wrong" reply scores 60, which is not far below the pass mark, despite containing a factual error. The critical-criterion check fails it immediately, which is the correct outcome for a support reply. Comparing Two Prompt Versions With a Rubric When comparing two prompt versions, you can score each version's outputs separately and compare average scores per criterion, or ask a judge to compare two outputs for the same input directly, criterion by criterion:

Compare reply A and reply B using the rubric in . For each criterion, state which reply is better, or "tie", with one sentence of evidence. Return only JSON: {"accuracy": {"winner": "A", "B", or "tie", "evidence": "..."}, ...} Pairwise judgments are often more consistent than absolute scores, because comparing is easier than scoring. As explained in Lesson 69, run each comparison twice with the order swapped, and accept a winner only when both runs agree, to neutralize position bias.

Per-criterion results show where a version is better: a new prompt may improve tone while making answers less complete, which a single overall winner would hide.

Checking the Judge Against People A model judge is itself a prompt that can be wrong or biased: it may prefer longer answers, overlook errors, or score its own model's outputs more generously. Before relying on it, calibrate it against human scores.

Measuring Agreement Have people score a sample of outputs with the same rubric, then compare:

def agreement(judge: list[int], human: list[int]) -> tuple[float, float]: """Return (exact agreement, agreement within one level).""" pairs = list(zip(judge, human)) exact = sum(j == h for j, h in pairs) / len(pairs) within_one = sum(abs(j - h) <= 1 for j, h in pairs) / len(pairs) return round(exact, 2), round(within_one, 2)

judge_scores = [3, 2, 3, 1, 2, 3, 3, 2, 1, 3] human_scores = [3, 2, 2, 1, 3, 3, 1, 2, 1, 3] print(agreement(judge_scores, human_scores)) print([i for i, (j, h) in enumerate(zip(judge_scores, human_scores), start=1) if abs(j - h) > 1]) Output:

(0.7, 0.9) [7] Code Explanation Exact agreement is the share of outputs where judge and human gave the same score: 7 of 10. Within-one agreement counts near misses as agreement: 9 of 10. Large disagreements are listed: output 7, where the judge gave 3 and the human gave 1, is the most important case to read. Using the Results Read every large disagreement. It usually reveals an unclear descriptor or a judge weakness, such as missing an invented promise. Improve the descriptors or the judge prompt, and measure again. Decide what level of agreement is acceptable for your use. For high-stakes decisions, keep humans involved regardless. More advanced agreement statistics, such as Cohen's kappa, adjust for agreement that would happen by chance. Simple exact and within-one agreement, plus reading disagreements, is a practical starting point.

Common Mistakes Mistake 1: Criteria Without Descriptors What it is: "Rate accuracy from 1 to 5."

Why it happens: The criterion name seems self-explanatory.

What it causes: Inconsistent scores between judges and between runs.

How to fix it: Describe what each level looks like for each criterion.

Mistake 2: Too Many Levels What it is: Scales from 1 to 10 for subjective qualities.

Why it happens: More levels seem more precise.

What it causes: Judges cannot distinguish adjacent levels consistently.

How to fix it: Use three or four clearly described levels.

Mistake 3: Averages That Hide Critical Failures What it is: Combining all criteria into one weighted score.

Why it happens: One number is convenient.

What it causes: Wrong but polished outputs pass.

How to fix it: Mark critical criteria with minimum scores that override the total.

Mistake 4: Scores Without Evidence What it is: Asking only for numbers.

Why it happens: Numbers are easy to parse.

What it causes: Scores that cannot be reviewed or explained.

How to fix it: Require evidence quotes before each score.

Mistake 5: An Uncalibrated Judge What it is: Using a model judge without comparing it to human scores.

Why it happens: The judge's explanations sound reasonable.

What it causes: Evaluation results that reflect the judge's biases, not real quality.

How to fix it: Measure agreement on a human-scored sample, and read disagreements.

Best Practices Choose a few criteria that matter for the task. Four or five is usually enough.

Write observable descriptors for every level. Three or four levels.

Weight criteria and mark critical ones. Critical failures override the total.

Ask for evidence before scores, criterion by criterion. Reduce halo effects and enable review.

Validate and combine scores in code. Exact, consistent rules.

Compare prompt versions per criterion. See where each version is better or worse.

Calibrate the judge against people. And re-check when the judge prompt or model changes.

Try It Yourself Exercise 1: Write a Rubric Write a rubric with four criteria and three levels for summaries of course announcements. Include descriptors for every level and mark one criterion as critical. Score five summaries yourself using the rubric.

Exercise 2: Run a Model Judge Use the judge prompt with your rubric on the same five summaries. Validate the scores with weighted_score() and apply verdict(). Compare the judge's scores with yours using agreement(), and read every disagreement larger than one level.

Exercise 3: Pairwise Comparison Create two prompt versions for the same task and generate outputs for ten inputs. Compare each pair with the pairwise rubric prompt, run each comparison twice with the order swapped, and count wins per criterion. Decide which version is better, and on which criteria.