Three Different Signals, Three Different Questions

Once an evaluation dataset exists (Lesson 6), running it through a model produces raw results — but raw results are not yet insight. Three specific metrics turn results into something you can track over time, compare across versions, and use to decide whether a system is production-ready:

• Accuracy answers: of the cases with a known correct answer, how many did the system get right?
• Consistency answers: if I ask the exact same question multiple times, how often do I get the same answer?
• Failure rate answers: how often does the system fail outright — an exception, a malformed response, a timeout — regardless of whether the content was correct?

These are distinct because a system can score well on one while failing badly on another. A system that is 95% accurate but only 60% consistent is unreliable in a way accuracy alone hides: it means the 95% figure came from a single run, and a different run on the same inputs could produce a meaningfully different number. A system that is both accurate and consistent but has a 10% failure rate (crashes or empty responses on one input in ten) still fails one in ten real user requests. Production readiness requires looking at all three together, not any single number in isolation.

Measuring Accuracy

Accuracy, for a classification-style task like the ticket-triage dataset from Lesson 6, is the simplest of the three: the fraction of examples where the system's output matches the expected label.

def compute_accuracy(results: list[dict]) -> float:
    """Each result dict has 'expected_category' and 'predicted_category'."""
    if not results:
        return 0.0
    correct = sum(
        1 for r in results if r["predicted_category"] == r["expected_category"]
    )
    return correct / len(results)


def test_compute_accuracy_basic():
    results = [
        {"expected_category": "billing", "predicted_category": "billing"},
        {"expected_category": "technical", "predicted_category": "account"},
        {"expected_category": "account", "predicted_category": "account"},
        {"expected_category": "other", "predicted_category": "other"},
    ]
    accuracy = compute_accuracy(results)
    assert accuracy == 0.75
    print("PASS: compute_accuracy returns the fraction of correct predictions")


def test_compute_accuracy_handles_empty_results():
    assert compute_accuracy([]) == 0.0
    print("PASS: compute_accuracy handles an empty results list without dividing by zero")

compute_accuracy is itself a piece of deterministic code — the same kind this whole unit has been building test coverage for — because a bug in the accuracy calculation (for example, an off-by-one in the denominator, or comparing the wrong two fields) would silently produce a misleading metric that no amount of dataset quality could fix. The empty-list guard matters in practice: a filtering step upstream that accidentally drops every record should surface as an obvious bug, not as a ZeroDivisionError or, worse, a wrong number.

Accuracy alone can be misleading when categories are imbalanced. If 80% of real tickets are technical, a system that always predicts technical scores 80% accuracy while being useless. A confusion matrix — a breakdown of predicted-versus-expected category counts — reveals this kind of failure that a single accuracy number hides.

from collections import defaultdict


def build_confusion_matrix(results: list[dict]) -> dict:
    matrix = defaultdict(lambda: defaultdict(int))
    for r in results:
        matrix[r["expected_category"]][r["predicted_category"]] += 1
    return {k: dict(v) for k, v in matrix.items()}


def test_build_confusion_matrix_tracks_misclassifications():
    results = [
        {"expected_category": "technical", "predicted_category": "billing"},
        {"expected_category": "technical", "predicted_category": "technical"},
        {"expected_category": "billing", "predicted_category": "billing"},
    ]
    matrix = build_confusion_matrix(results)
    assert matrix["technical"]["billing"] == 1
    assert matrix["technical"]["technical"] == 1
    assert matrix["billing"]["billing"] == 1
    print("PASS: build_confusion_matrix tracks which categories are confused with which")

The confusion matrix shows which categories get mixed up with which others — in this example, one technical ticket was misclassified as billing. Knowing this specific failure pattern is far more actionable than knowing an overall accuracy percentage, because it points directly at where prompt or schema improvements should focus.

Measuring Consistency

Consistency requires running the same input through the model multiple times and checking how often the outputs agree with each other — not with a ground-truth label, but with each other. This matters because temperature, sampling, and model updates all introduce variability that a single-run accuracy score cannot detect.

from collections import Counter


def compute_consistency(repeated_outputs: list[str]) -> float:
    """Given N outputs for the *same* input, return the fraction matching the mode."""
    if not repeated_outputs:
        return 0.0
    counts = Counter(repeated_outputs)
    most_common_count = counts.most_common(1)[0][1]
    return most_common_count / len(repeated_outputs)


def test_compute_consistency_fully_consistent():
    outputs = ["billing", "billing", "billing", "billing"]
    assert compute_consistency(outputs) == 1.0
    print("PASS: compute_consistency returns 1.0 when all outputs agree")


def test_compute_consistency_partial_agreement():
    outputs = ["billing", "billing", "account", "billing"]
    assert compute_consistency(outputs) == 0.75
    print("PASS: compute_consistency returns the fraction matching the majority answer")

compute_consistency treats the most frequent output as the "consensus" answer and reports what fraction of the repeated runs agreed with it — it deliberately does not need to know the correct answer, only whether the system agrees with itself. A consistency score run across an entire dataset (repeating each example some fixed number of times, for example 5) gives an average consistency figure that reveals how much of your accuracy number might shift on a re-run purely due to randomness.

def average_consistency_across_dataset(per_example_outputs: dict) -> float:
    """per_example_outputs maps example_id -> list of repeated outputs."""
    if not per_example_outputs:
        return 0.0
    scores = [compute_consistency(outputs) for outputs in per_example_outputs.values()]
    return sum(scores) / len(scores)


def test_average_consistency_across_dataset():
    per_example_outputs = {
        "triage-001": ["billing", "billing", "billing"],
        "triage-002": ["technical", "account", "technical"],
    }
    avg = average_consistency_across_dataset(per_example_outputs)
    assert round(avg, 4) == round((1.0 + (2 / 3)) / 2, 4)
    print("PASS: average_consistency_across_dataset averages per-example consistency scores")

Note: Lowering temperature toward 0 generally increases consistency for classification-style tasks, but does not guarantee perfect determinism — infrastructure-level nondeterminism can still produce different outputs for identical inputs on some models and configurations.

Measuring Failure Rate

Failure rate is distinct from inaccuracy: an inaccurate answer is a wrong-but-valid answer ("billing" when the truth was "technical"), while a failure is the absence of a usable answer at all — an exception, a timeout, an empty string, or output that does not parse as one of the valid categories.

def compute_failure_rate(raw_results: list[dict]) -> float:
    """Each raw_result has 'predicted_category', which may be None on failure."""
    if not raw_results:
        return 0.0
    failures = sum(
        1 for r in raw_results
        if r["predicted_category"] is None or not is_valid_category(r["predicted_category"])
    )
    return failures / len(raw_results)


def test_compute_failure_rate_counts_none_and_invalid_predictions():
    raw_results = [
        {"predicted_category": "billing"},
        {"predicted_category": None},          # exception during the call
        {"predicted_category": "not_a_label"},  # model returned something invalid
        {"predicted_category": "technical"},
    ]
    rate = compute_failure_rate(raw_results)
    assert rate == 0.5
    print("PASS: compute_failure_rate counts both missing and invalid predictions as failures")

This function relies on is_valid_category from Lesson 6, reinforcing why defining the valid output space explicitly, as its own testable function, pays off here: failure detection and accuracy scoring both depend on a shared, unambiguous notion of what counts as a legitimate answer at all. In a real pipeline, predicted_category becomes None specifically because the calling code caught an exception (a timeout, a malformed structured-output response, an API error) and recorded the failure rather than letting it crash the whole evaluation run — which is itself a piece of application logic worth unit testing with the mocking techniques from Lesson 3.

Tracking All Three Over Time

None of these three metrics is meaningful as a single snapshot; their value comes from tracking them across evaluation runs — after every meaningful prompt change, schema change, or model version bump — and watching for regressions. A simple record format makes this tracking straightforward to build on top of the functions above:

def summarize_evaluation_run(run_id: str, results: list[dict], raw_results: list[dict],
                              per_example_outputs: dict) -> dict:
    return {
        "run_id": run_id,
        "accuracy": compute_accuracy(results),
        "consistency": average_consistency_across_dataset(per_example_outputs),
        "failure_rate": compute_failure_rate(raw_results),
    }

Storing the output of summarize_evaluation_run for every run (appended to a log file, a database table, or a JSON Lines history) is what makes the next lesson's regression testing possible: comparing today's numbers against a known-good baseline requires that baseline to have been recorded somewhere in exactly this shape.

Common Mistakes

• Reporting only accuracy and ignoring consistency. A high accuracy figure from a single run can be an artifact of favorable randomness; without a consistency measurement, you cannot tell whether a re-run would reproduce the same score.
• Counting a wrong-but-valid answer the same as an outright failure. Conflating "the model was wrong" with "the system crashed" hides two very different engineering problems — one is a prompt or model quality issue, the other is often a bug or missing error handling in your code.
• Computing metrics on filtered or partial result sets without noticing. A bug that silently drops failed examples before computing accuracy will report an artificially high accuracy, since the hardest, most-likely-to-fail cases were never counted at all.

Best Practices

• Always report accuracy, consistency, and failure rate together, since each one can look good while another hides a real problem.
• Repeat at least a subset of dataset examples multiple times to measure consistency directly, rather than assuming a single run's accuracy is representative.
• Persist every evaluation run's metrics in a consistent, comparable format, tagged with what changed (prompt version, model version, schema version), so regressions can be detected automatically rather than noticed anecdotally.