Evaluating Accuracy, Relevance, and Completeness

Three Questions About Every Answer For most question-answering, summarization, and support tasks, three questions capture whether an output is useful:

Dimension Question Failure Example Accuracy Is what it says correct? "Refunds are available within 14 days" when the policy says 7 Relevance Does it address what was asked, and only that? A refund question answered with information about certificates Completeness Does it cover everything required? The answer gives the refund window but omits the 20% progress condition These dimensions are independent. An answer can be accurate but incomplete, complete but inaccurate, or accurate and complete but padded with irrelevant information. Measuring them separately shows what kind of problem a prompt has, which tells you how to fix it.

Measured Problem Typical Fix Low accuracy Better context, grounding rules, retrieval (Lessons 104–106) Low relevance Clearer task focus, scope rules, length limits (Lesson 14) Low completeness Explicit checklist of required points, better retrieval, longer limit if needed Accuracy Defining Accuracy An output is accurate when its factual statements agree with a source of truth: a reference answer, a document, a database, or a verified fact list. Without a source of truth, accuracy cannot be measured reliably; you can only measure whether outputs sound correct.

Measuring Accuracy by Task Type Task Type Method Labels, extracted fields Exact match with expected values (Lesson 114) Numbers and dates Compare values in code, after normalizing formats Free-text answers Check claims against reference facts or sources RAG answers Claim-level support checks against retrieved sources (Lesson 106) Claim-Level Accuracy For free-text answers, split the answer into claims and check each one against the reference facts:

Reference facts for "Can I get a refund?": F1. Refunds are available within 7 days of purchase. F2. Refunds require less than 20% course completion. F3. Approved refunds take 5–7 business days.

For each sentence in , label it:

• CORRECT: agrees with a reference fact (give the fact id)
• INCORRECT: contradicts a reference fact (give the fact id)
• UNVERIFIABLE: not covered by the reference facts Return only JSON. Correct, Incorrect, and Unverifiable Distinguishing incorrect from unverifiable is important:

Incorrect claims contradict the truth. They are always failures. Unverifiable claims are not covered by the reference facts. They may be harmless ("Thanks for reaching out") or may be invented details ("Refunds are faster for premium members"). In grounded applications, unverifiable factual claims should be treated as failures too (Lesson 106). Accuracy Metrics Metric Calculation Answer-level accuracy Share of answers with no incorrect claims Claim-level accuracy Correct claims ÷ all factual claims Error rate Share of answers with at least one incorrect claim Answer-level accuracy is strict and matches user experience: one wrong fact can make an entire answer harmful.

Completeness Defining Completeness An output is complete when it covers every required point. Required points come from the question and the task: for a refund question, the refund window, the progress condition, and the processing time might all be required.

Required Points With Aliases Write the required points for each test case (Lesson 115), with acceptable ways of expressing them. Simple coverage can then be checked in code:

import re

REQUIRED_POINTS = { "refund_window": ["7 days", "seven days", "within a week"], "progress_limit": ["20%", "20 percent", "twenty percent"], "refund_time": ["5 to 7 business days", "5-7 business days"], }

def sentences(text: str) -> list[str]: return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()]

def evaluate_answer(answer: str, required: dict[str, list[str]] = REQUIRED_POINTS) -> dict: lowered = answer.lower() covered = {point: any(alias in lowered for alias in aliases) for point, aliases in required.items()} all_aliases = [alias for aliases in required.values() for alias in aliases] unmatched = [s for s in sentences(answer) if not any(a in s.lower() for a in all_aliases)] return { "completeness": round(sum(covered.values()) / len(covered), 2), "missing": [point for point, ok in covered.items() if not ok], "sentences_without_required_point": unmatched, }

answer = ("You can request a refund within 7 days of purchase. " "Approved refunds take 5 to 7 business days to reach you. " "Our Django course was recently updated with new projects.") print(evaluate_answer(answer)) Output:

{'completeness': 0.67, 'missing': ['progress_limit'], 'sentences_without_required_point': ['Our Django course was recently updated with new projects.']} Code Explanation covered For each required point, checks whether any of its aliases appears in the lowercased answer.

completeness The share of required points covered: two of three, or 0.67.

missing Lists the points not covered. The answer never mentions the 20% progress condition, which could lead a student who has completed half the course to request a refund they cannot get.

sentences_without_required_point Lists sentences that do not mention any required point. This is a simple signal for relevance, explained below.

Limits of Alias Matching Alias matching is fast and transparent, but it misses paraphrases such as "you need to have finished less than a fifth of the course". Extend aliases as you find valid paraphrases, or use a model judge for semantic coverage:

For each required point in , decide whether clearly communicates it, even in different words. Return only JSON: {"point_id": true or false, ...} Use code checks for speed and consistency, and a judge for points that are often expressed in different ways. Calibrate the judge against human labels (Lesson 116).

Relevance Defining Relevance An output is relevant when it addresses the question that was asked and stays on that topic. Two kinds of relevance problems are common:

Problem Example Off-target answer A question about refunds answered with enrollment instructions Irrelevant additions A correct refund answer followed by unrelated course news Why Irrelevant Content Matters Irrelevant sentences are not harmless padding:

They make important information harder to find. They add length and cost. They are where unsupported claims often appear, because the model is no longer working from the question's sources. In the example above, "Our Django course was recently updated with new projects" answers nothing the student asked, and it may not be supported by any source.

Measuring Relevance Method How Unmatched sentences Sentences that mention no required point, as in the code above; review them Model judge per sentence Label each sentence RELEVANT or NOT_RELEVANT to the question Off-target check A judge decides whether the answer addresses the question at all The unmatched-sentence signal is a starting point, not a verdict: polite phrases such as "Thanks for asking" are harmless. Decide which kinds of extra content are acceptable for your application, and state that in the prompt and the evaluation.

Putting the Three Together An Evaluation Record For each test case, record all three dimensions:

{ "id": "refund-01", "accuracy": {"incorrect_claims": 0, "unverifiable_claims": 1}, "completeness": {"score": 0.67, "missing": ["progress_limit"]}, "relevance": {"off_target": false, "irrelevant_sentences": 1} } Interpreting Combined Results Accuracy Completeness Relevance Diagnosis High Low High Prompt or retrieval misses required information; add a checklist of points or improve retrieval Low High High The answer covers the right topics but states them wrongly; improve grounding and sources High High Low Correct and complete, but padded; add scope and length rules High High High Good; monitor for regressions (Lesson 118) Trade-Offs Improving one dimension can hurt another:

Asking for shorter answers improves relevance but can reduce completeness. Asking for complete answers can lead to padding and unsupported additions. Strict grounding improves accuracy but can make answers incomplete when sources are partial (Lesson 105). Track all three dimensions when changing a prompt, so an improvement in one is not hiding a loss in another (Lesson 66).

Common Mistakes Mistake 1: One Combined "Quality" Score What it is: Rating answers with a single overall score.

Why it happens: It is simpler to report.

What it causes: You cannot tell whether answers are wrong, incomplete, or unfocused.

How to fix it: Measure accuracy, completeness, and relevance separately.

Mistake 2: Accuracy Without a Source of Truth What it is: Judging whether answers "seem correct".

Why it happens: Reference facts take effort to write.

What it causes: Plausible but wrong answers are rated as accurate.

How to fix it: Define reference facts or sources for each test case.

Mistake 3: Ignoring Unverifiable Claims What it is: Counting only claims that contradict the reference.

Why it happens: Unverifiable claims are not obviously wrong.

What it causes: Invented details pass evaluation.

How to fix it: Track unverifiable claims separately, and treat factual ones as failures in grounded systems.

Mistake 4: Completeness Without Required Points What it is: Asking whether an answer is "complete" without defining what it must contain.

Why it happens: Completeness seems intuitive.

What it causes: Inconsistent judgments.

How to fix it: Write required points, with aliases, for each test case.

Mistake 5: Rewarding Length What it is: Treating longer answers as more complete.

Why it happens: Longer answers often mention more topics.

What it causes: Prompts drift toward padded, less relevant answers.

How to fix it: Measure relevance alongside completeness, and keep length limits.

Best Practices Measure accuracy, completeness, and relevance separately. Each points to different fixes.

Ground accuracy in a source of truth. Reference facts, documents, or databases.

Classify claims as correct, incorrect, or unverifiable. Treat unverifiable facts with suspicion.

Define required points with aliases. Check coverage in code, use judges for paraphrases.

Flag content that matches no required point. Review it for relevance and unsupported claims.

Watch the trade-offs. Improvements in one dimension must not hide losses in another.

Try It Yourself Exercise 1: Required Points For five questions about your platform's policies, write required points with at least two aliases each. Run evaluate_answer() on answers from your RAG prompt, and list the missing points for each.

Exercise 2: Claim Labels For three answers, write reference facts and use the claim-labeling prompt to classify each sentence as correct, incorrect, or unverifiable. Check the labels yourself, and calculate answer-level accuracy.

Exercise 3: Diagnose and Fix Using your results from Exercises 1 and 2, find the dimension with the most problems. Make one targeted prompt change for that dimension, re-run the evaluation, and check whether the other two dimensions changed.