Evaluating Semantic Search Quality

Every design decision in this unit — which embedding model, what chunk size, whether to add metadata filtering, which similarity threshold to use — is ultimately a guess unless it is checked against measured results. Unit 13 introduced the general methodology for evaluating and improving LLM-based systems: build a labeled dataset, run the system against it, score the output with a grader, and track the score as changes are made. This lesson applies that exact methodology to retrieval specifically, using metrics designed for ranked search results rather than the free-text grading covered in Unit 13.

Why "It Looks Right" Is Not Evaluation

A developer testing a search feature by typing in a few queries and eyeballing the results will reliably miss two kinds of problems: queries that are subtly worse than they appear (a relevant document ranked 4th instead of 1st looks "fine" at a glance but represents a real quality gap), and regressions introduced by a later change (a new chunking strategy improves some queries and quietly worsens others, and manual spot-checking rarely catches both directions at once). A measured evaluation, run the same way every time against the same fixed dataset, catches both.

Building a Retrieval Evaluation Dataset

Following the same pattern Unit 13 used for building eval datasets, a retrieval eval dataset is a set of realistic queries, each paired with the set of documents that are actually relevant to it — established by a human, not by the system being evaluated (grading a system using its own opinion of what is relevant is circular).

from dataclasses import dataclass


@dataclass
class RetrievalExample:
    query: str
    relevant_doc_ids: set[str]   # ground truth, labeled independently of the search system


eval_dataset = [
    RetrievalExample(
        query="how do I get a refund on a broken item",
        relevant_doc_ids={"returns-policy", "damaged-goods-process"},
    ),
    RetrievalExample(
        query="when will my order arrive",
        relevant_doc_ids={"shipping-times", "order-tracking"},
    ),
    RetrievalExample(
        query="can I change my shipping address after ordering",
        relevant_doc_ids={"order-tracking", "shipping-times"},
    ),
]

Why must relevant_doc_ids come from independent human judgment? The entire point of evaluation is to measure whether the system finds what a real user would consider relevant. If "relevant" is instead defined as "whatever the current system already returns," the evaluation can never detect that the system is missing genuinely relevant documents — it would only ever confirm the system agrees with itself. In practice, this dataset is built by having someone (a domain expert, or the developer acting carefully) review each query against the full document set once, independent of any particular search run, and record which documents actually answer it.

Core Retrieval Metrics

Three metrics, all computed from the same basic input — a ranked list of retrieved document IDs compared against a set of known-relevant IDs — cover most practical evaluation needs.

Precision@k — of the top k results returned, what fraction are actually relevant?

def precision_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    top_k = retrieved_ids[:k]
    if not top_k:
        return 0.0
    relevant_in_top_k = sum(1 for doc_id in top_k if doc_id in relevant_ids)
    return relevant_in_top_k / len(top_k)

Recall@k — of all the documents that are actually relevant, what fraction were found in the top k results?

def recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    if not relevant_ids:
        return 0.0
    top_k = retrieved_ids[:k]
    relevant_in_top_k = sum(1 for doc_id in top_k if doc_id in relevant_ids)
    return relevant_in_top_k / len(relevant_ids)

Why track both, instead of just one? They measure different failure modes. A search that returns one correct result and nine irrelevant ones in its top 10 has poor precision but might have fine recall for a query with only one relevant document. A search that returns five correct results but misses five other equally relevant ones has fine precision but poor recall. A system that ranks well on precision but poorly on recall is systematically failing to surface some relevant content — often a signal to revisit chunking (Lesson 8) or check whether relevant content was indexed at all.

Mean Reciprocal Rank (MRR) — for queries with one clearly best answer, how high up the ranking does the first relevant result appear?

def reciprocal_rank(retrieved_ids: list[str], relevant_ids: set[str]) -> float:
    for position, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in relevant_ids:
            return 1.0 / position
    return 0.0


def mean_reciprocal_rank(all_retrieved: list[list[str]], all_relevant: list[set[str]]) -> float:
    scores = [reciprocal_rank(r, rel) for r, rel in zip(all_retrieved, all_relevant)]
    return sum(scores) / len(scores) if scores else 0.0

Why does MRR matter separately from precision and recall? For search interfaces where users typically only look at the first result or two (a "did you mean" style suggestion, an autocomplete), what matters most is how quickly the first relevant result appears, not the overall composition of the top 10. A reciprocal rank of 1.0 means the first result was relevant; 0.5 means the first relevant result was in position 2; a score approaching 0 means it took many results to find one relevant document, or none appeared at all.

Running the Evaluation

def test_retrieval_metrics_with_fake_results():
    example = RetrievalExample(
        query="how do I get a refund on a broken item",
        relevant_doc_ids={"returns-policy", "damaged-goods-process"},
    )
    # A fake, hand-constructed ranked result list standing in for a real search call.
    retrieved = ["damaged-goods-process", "shipping-times", "returns-policy", "order-tracking"]

    p_at_2 = precision_at_k(retrieved, example.relevant_doc_ids, k=2)
    r_at_2 = recall_at_k(retrieved, example.relevant_doc_ids, k=2)
    rr = reciprocal_rank(retrieved, example.relevant_doc_ids)

    assert p_at_2 == 0.5      # 1 of the top 2 results is relevant
    assert r_at_2 == 0.5      # 1 of 2 relevant docs found in the top 2
    assert rr == 1.0          # the very first result is relevant

    print("PASS: retrieval metrics computed correctly against fixed fake results")


test_retrieval_metrics_with_fake_results()

This test does not call the search engine at all — it works directly against a hand-written retrieved list, which is the right level to test the metric functions themselves: their correctness should not depend on what any particular embedding model actually returns. A separate evaluation run — shown next — is what exercises the real search engine against the full eval dataset.

def evaluate_search_engine(engine, dataset: list[RetrievalExample], k: int = 5) -> dict:
    """Run the full eval dataset through a real search engine and
    aggregate metrics. `engine` must expose `.search(query, top_k)`
    returning a list of (document, score) pairs whose documents have
    a `.doc_id` attribute (matching SemanticSearchEngine from Lesson 5).
    """
    precisions, recalls, reciprocal_ranks = [], [], []

    for example in dataset:
        results = engine.search(example.query, top_k=k)
        retrieved_ids = [doc.doc_id for doc, _ in results]

        precisions.append(precision_at_k(retrieved_ids, example.relevant_doc_ids, k))
        recalls.append(recall_at_k(retrieved_ids, example.relevant_doc_ids, k))
        reciprocal_ranks.append(reciprocal_rank(retrieved_ids, example.relevant_doc_ids))

    return {
        "avg_precision_at_k": sum(precisions) / len(precisions),
        "avg_recall_at_k": sum(recalls) / len(recalls),
        "mean_reciprocal_rank": sum(reciprocal_ranks) / len(reciprocal_ranks),
        "num_queries": len(dataset),
    }

Running evaluate_search_engine against the same fixed eval_dataset before and after a change — a new chunking strategy, a different embedding model, an added metadata filter — produces a direct, numeric before/after comparison. This is the same core discipline Unit 13 established for grading generated text: fix the dataset, run the system, record the score, change one thing at a time, and re-measure.

When Metrics Alone Are Not Enough: Using a Grader

Precision, recall, and MRR require a labeled set of "correct" document IDs, which works well when relevance is fairly clear-cut. Some queries have fuzzier relevance — a document might be partially relevant, or relevant in a way a strict ID-matching label does not capture. For these cases, Unit 13's grader pattern (using a capable model to judge quality against a rubric) can be adapted to retrieval by asking the grader to judge whether a retrieved chunk actually helps answer the query, rather than only asking whether its ID matches a predetermined label.

def grade_relevance(client, query: str, retrieved_text: str) -> bool:
    """Ask a model to judge whether a retrieved chunk is relevant to a query.

    This mirrors Unit 13's grader pattern applied specifically to
    retrieval: the grader answers one narrow, binary question rather
    than producing open-ended commentary, which keeps grading
    consistent and easy to aggregate across many examples.
    """
    prompt = (
        f"Query: {query}\n\n"
        f"Retrieved passage:\n{retrieved_text}\n\n"
        "Does this passage contain information that helps answer the query? "
        "Answer with exactly one word: yes or no."
    )
    response = client.responses.create(model="gpt-5.6-terra", input=prompt)
    return response.output_text.strip().lower().startswith("y")

Note: The exact response field (output_text) and request shape shown here follow the pattern introduced in Unit 1; confirm current field names against the official API reference, since response object structure can change between SDK versions.

Why use a binary yes/no question rather than asking for a relevance score out of 10? Unit 13 established that narrow, well-defined grading questions produce more consistent results than open-ended scoring, because a model asked for a precise numeric score tends to give inconsistent numbers for similar inputs, while a binary judgment is easier for both a model and a human reviewer to apply consistently. A grader like this is best used to spot-check queries that the ID-based metrics above cannot label cleanly (novel queries without a pre-built ground truth set), not as a wholesale replacement for the labeled dataset — a fixed labeled dataset remains cheaper, faster, and fully reproducible for tracking changes over time.

Common Mistakes

• Evaluating with only a handful of queries chosen informally. A tiny, non-representative eval set can show an improvement on the queries it happens to contain while missing regressions elsewhere; Unit 13's guidance on building a sufficiently sized, representative dataset applies here without modification.
• Deriving "relevant" labels from the system's own current output. This makes the evaluation circular and unable to detect missed, genuinely relevant documents, as explained above.
• **Changing more than one variable (chunk size and embedding model and similarity threshold) between evaluation runs.** This makes it impossible to attribute a metric change to a specific cause — change one variable at a time, exactly as Unit 13 recommends for iterative improvement.

Best Practices

• Build the eval dataset once, keep it under version control, and reuse it for every future change — a stable dataset is what makes before/after comparisons meaningful.
• Report precision, recall, and MRR together, not a single blended number, since each exposes a different kind of failure, as shown above.
• Reserve model-based grading for cases the labeled dataset cannot cleanly cover, using the narrow binary-question pattern from Unit 13 rather than open-ended scoring.