What Each Approach Actually Measures

Every metric built in this unit so far — accuracy, consistency, failure rate, regression comparisons — comes from automated evaluation: code that compares a model's output against a fixed expected answer or a fixed rule, with no person in the loop for each individual judgment. Automated evaluation is fast, cheap at scale, and perfectly repeatable, which is exactly why it fits naturally into unit tests, CI pipelines, and regression gates.

Human evaluation replaces the automated comparison with a person reading the model's output and judging it — often on dimensions that are difficult or impossible to reduce to a rule: Is this response actually helpful, not just technically correct? Does the tone match the brand? Is this medical or legal information stated responsibly? Would a real customer find this satisfying? These questions require judgment that a fixed rubric or an exact-match comparison cannot fully capture, which is precisely the gap human evaluation exists to fill.

Neither approach is strictly better — they answer different questions, at different cost, at different speed, and with different failure modes. Production systems that rely on evaluation seriously tend to use both, deliberately, for the parts of the problem each is suited to.

Where Automated Evaluation Is the Right Tool

Automated evaluation is the correct choice whenever "correct" can be defined precisely enough for code to check it. The ticket-triage classification task from Lesson 6 is a clean example: a fixed set of valid categories, an unambiguous expected label for each dataset example, and a comparison that a computer can perform instantly and identically every time.

def automated_grade(predicted: str, expected: str) -> bool:
    return predicted.strip().lower() == expected.strip().lower()


def test_automated_grade_exact_match():
    assert automated_grade("Billing", "billing") is True
    print("PASS: automated_grade normalizes case and whitespace before comparing")


def test_automated_grade_detects_mismatch():
    assert automated_grade("technical", "billing") is False
    print("PASS: automated_grade correctly flags a mismatched label")

Automated grading like this scales to thousands of examples run every night in CI at essentially zero marginal cost, and it produces the exact same verdict every time it is run on the same inputs — a property human graders cannot guarantee, since two different people (or the same person on two different days) can reasonably disagree about a borderline case. This reliability is what makes automated evaluation the right foundation for regression gates (Lesson 8): a gate that sometimes disagrees with itself is not trustworthy as a gate.

Where Automated Evaluation Falls Short

Automated grading struggles as soon as "correct" stops being a fixed, checkable fact and becomes a matter of quality or judgment. Consider grading whether a customer-support response is appropriately empathetic — there is no fixed string to match, no schema to validate against, and a rule like "contains the word 'sorry'" would be both easy to game and a poor proxy for genuine empathy.

def naive_empathy_check(response_text: str) -> bool:
    empathy_keywords = ["sorry", "understand", "apologize", "frustrating"]
    lowered = response_text.lower()
    return any(keyword in lowered for keyword in empathy_keywords)


def test_naive_empathy_check_is_easily_gamed():
    # This response contains the keyword but is dismissive and unhelpful.
    fake_response = "Sorry, that's not our problem. Read the manual."
    assert naive_empathy_check(fake_response) is True
    print("PASS (illustrates the limitation): keyword presence does not imply genuine empathy")

This test intentionally demonstrates the failure mode rather than a success: naive_empathy_check returns True for a response that is clearly unhelpful, simply because it happens to contain the word "sorry." This is the core limitation of automated evaluation for subjective qualities — any rule simple enough for code to check reliably is usually simple enough for a model (or a person gaming a metric) to satisfy without actually achieving the underlying goal. Model-graded evaluation (using a second model call as a grader, covered in Unit 13) narrows this gap somewhat by using judgment rather than keyword matching, but it introduces its own failure mode: the grading model can share the same blind spots as the model being graded, or can itself be miscalibrated in ways that go unnoticed without some human oversight.

What Human Evaluation Adds

Human evaluation is the right tool specifically for the qualities automated grading cannot reliably approximate: genuine helpfulness, tone and brand alignment, nuanced safety judgments, and catching failure modes nobody anticipated well enough to write a rule for in the first place. A simple, practical structure for human evaluation is a rating rubric applied by reviewers to a sample of real or representative outputs.

from dataclasses import dataclass


@dataclass
class HumanReviewRecord:
    example_id: str
    reviewer: str
    helpfulness_score: int   # 1-5
    tone_appropriate: bool
    notes: str


def validate_human_review(record: HumanReviewRecord) -> None:
    if not (1 <= record.helpfulness_score <= 5):
        raise ValueError("helpfulness_score must be between 1 and 5")
    if not record.reviewer.strip():
        raise ValueError("reviewer must be identified")


def test_validate_human_review_accepts_well_formed_record():
    record = HumanReviewRecord(
        example_id="support-042",
        reviewer="reviewer_a",
        helpfulness_score=4,
        tone_appropriate=True,
        notes="Clear and polite, but missed the refund timeline question.",
    )
    validate_human_review(record)  # should not raise
    print("PASS: validate_human_review accepts a properly filled-out review")


def test_validate_human_review_rejects_out_of_range_score():
    record = HumanReviewRecord(
        example_id="support-043",
        reviewer="reviewer_a",
        helpfulness_score=7,
        tone_appropriate=True,
        notes="",
    )
    try:
        validate_human_review(record)
        raised = False
    except ValueError:
        raised = True
    assert raised
    print("PASS: validate_human_review rejects an out-of-range helpfulness score")

Notice that even the process of collecting human evaluation benefits from ordinary software testing: validate_human_review is deterministic code that enforces data-quality rules on human input, exactly the same way ExtractedInvoice (Lesson 4) enforces rules on model input. The human judgment itself (the helpfulness_score a reviewer assigns) is not something code can test, but the structure around collecting that judgment absolutely is.

Measuring Agreement Among Human Reviewers

A single reviewer's opinion carries some amount of unavoidable subjectivity. When multiple reviewers rate the same examples, the degree to which they agree — inter-rater agreement — indicates how reliable the rubric itself is. Low agreement usually means the rubric's categories are ambiguous or under-specified, not that the reviewers are careless.

def compute_pairwise_agreement(reviewer_a_scores: list[int], reviewer_b_scores: list[int]) -> float:
    if len(reviewer_a_scores) != len(reviewer_b_scores):
        raise ValueError("Both reviewers must have scored the same number of examples")
    if not reviewer_a_scores:
        return 0.0

    exact_matches = sum(
        1 for a, b in zip(reviewer_a_scores, reviewer_b_scores) if a == b
    )
    return exact_matches / len(reviewer_a_scores)


def test_compute_pairwise_agreement_full_agreement():
    a = [4, 5, 3, 2]
    b = [4, 5, 3, 2]
    assert compute_pairwise_agreement(a, b) == 1.0
    print("PASS: compute_pairwise_agreement returns 1.0 for identical scores")


def test_compute_pairwise_agreement_partial_agreement():
    a = [4, 5, 3, 2]
    b = [4, 4, 3, 1]
    assert compute_pairwise_agreement(a, b) == 0.5
    print("PASS: compute_pairwise_agreement returns the fraction of exactly matching scores")

A low pairwise agreement score across many examples is a signal to revise the rubric — for example, replacing a vague instruction like "rate helpfulness 1-5" with concrete anchors for each score ("5 = fully resolves the customer's question with no follow-up needed") — rather than a signal to distrust the reviewers individually. This is directly analogous to the dataset-labeling discipline from Lesson 6: undocumented, ambiguous judgment calls degrade reliability whether the judge is a person labeling a dataset or a person scoring a live model output.

Combining Both in Practice

The most effective structure uses automated evaluation as the fast, cheap, always-on layer — running on every relevant change, gating regressions (Lesson 8) — and reserves human evaluation for periodic, deeper audits: a sample of real production outputs reviewed on a rubric monthly, or before a major prompt or model change ships, specifically to catch the qualities automated grading structurally cannot see. A useful additional practice is periodically checking automated grading against human judgment on the same examples, to catch cases where the automated proxy has quietly drifted from what actually matters to users.

Common Mistakes

• Relying solely on automated metrics for subjective qualities. A high automated score on a proxy metric like keyword presence can mask genuinely poor output quality, as the empathy example above demonstrates directly.
• Using human evaluation for everything, including checks a rule could handle. Human review is slow and expensive relative to automated grading; spending reviewer time on checks that a schema validator or exact-match comparison could perform instead wastes a scarce resource.
• Ignoring low inter-rater agreement. Treating a rubric's poor agreement scores as a reviewer-competence problem instead of a rubric-design problem prevents the actual fix (clarifying the rubric) from ever happening.

Best Practices

• Match the evaluation method to the question being asked: use automated grading for anything with a well-defined correct answer, and reserve human evaluation for genuinely subjective or safety-sensitive judgments.
• Give human reviewers a concrete, anchored rubric, not a vague scale, and measure inter-rater agreement to catch rubric ambiguity early.
• Periodically validate automated grading against human judgment on a shared sample of outputs, so an automated proxy metric that has drifted away from real quality gets caught rather than silently trusted indefinitely.