Why This Deserves Its Own Lesson

A RAG-based assistant fails silently in a specific and dangerous way: when the vector store doesn't contain a good answer to a question, the model can still produce a fluent, confident-sounding response drawn from its general training knowledge rather than the retrieved documents — and to a user, that response looks exactly like a properly grounded one. This is often worse than the model plainly saying "I don't know," because an ungrounded but confident answer erodes the entire premise of building a document-grounded assistant in the first place: trustworthiness. This lesson is about detecting and handling that failure mode deliberately, rather than hoping instructions alone prevent it.

The Three Distinct Failure Modes

It helps to separate what looks like one problem ("the assistant gave a bad answer") into three distinct causes, because each has a different fix:

1. No relevant documents exist at all. The knowledge base genuinely doesn't cover the topic the user is asking about.
2. Relevant documents exist but weren't retrieved. The right chunk is in the vector store, but similarity search didn't surface it — often due to a phrasing mismatch, poor chunking, or an overly narrow metadata filter (Lesson 5's "filter matching nothing" mistake is one cause of this).
3. Relevant chunks were retrieved, but the model ignored or misused them, generating an answer that isn't actually supported by the retrieved text even though the evidence was present in context.

Case 1 is a content gap — the fix is adding documents. Case 2 is a retrieval quality problem — the fixes are in Lesson 7 (document preparation) and Lesson 5 (filtering). Case 3 is a generation discipline problem — the fix is instructions and verification, covered below. Conflating these three during debugging leads to fixing the wrong layer of the system.

Instructing the Model to Acknowledge Gaps

The first and cheapest layer of defense is explicit instruction, already introduced in Lesson 4 and Lesson 6:

from openai import OpenAI

client = OpenAI()

GROUNDED_INSTRUCTIONS = (
    "You are a knowledge base assistant. Answer only using information "
    "retrieved via file search. If the retrieved documents do not contain "
    "enough information to answer confidently, respond with: "
    "\"I don't have enough information in the knowledge base to answer that.\" "
    "Do not use general knowledge to fill gaps, and do not guess."
)

response = client.responses.create(
    model="gpt-5.6-terra",
    instructions=GROUNDED_INSTRUCTIONS,
    input="What is our policy on parental leave in Germany specifically?",
    tools=[{"type": "file_search", "vector_store_ids": ["vs_hr_policies_68f2"]}],
)

print(response.output_text)

Prescribing an exact fallback phrase ("I don't have enough information...") rather than a vague instruction like "say you're not sure" is deliberate: a consistent, recognizable fallback string is something your application code can detect programmatically (shown next), which turns "the model was appropriately uncertain" into a signal your system can act on, rather than something buried in free-form prose that's hard to parse reliably.

Instructions alone are not a guarantee — models can still occasionally answer from general knowledge despite being told not to, particularly for well-known topics adjacent to what's in the documents. Instructions reduce the frequency of this failure; they don't eliminate it, which is why the additional layers below matter.

Detecting the No-Search-Happened Case

Sometimes the model answers without invoking file_search at all — a general knowledge question that doesn't clearly relate to the knowledge base might not trigger the tool. Detecting this from the structured output lets you flag it explicitly:

def was_file_search_invoked(response):
    return any(item.type == "file_search_call" for item in response.output)


def get_retrieved_file_count(response):
    count = 0
    for item in response.output:
        if item.type == "file_search_call":
            results = getattr(item, "results", None) or []
            count += len(results)
    return count

Note: The presence and structure of a results field on a file_search_call item, and whether an empty search is represented as zero results versus the item being absent entirely, are version-specific details — confirm current behavior against official documentation.

was_file_search_invoked checks whether any item in the response's output was a file_search_call at all, which tells you whether retrieval was attempted. get_retrieved_file_count goes a step further, counting how many actual results came back from any search calls that did happen. A response where was_file_search_invoked is False for a question that should clearly be answerable from the knowledge base is worth flagging for review — either the question fell outside the model's judgment of when to search, or something about the request configuration (wrong vector_store_ids, for instance) prevented it.

Detecting Low-Confidence or Empty Retrieval

Combining the fallback-phrase detection with the structured retrieval signal gives a more complete picture than either alone:

FALLBACK_PHRASE = "I don't have enough information in the knowledge base to answer that."


def classify_response(response):
    searched = was_file_search_invoked(response)
    result_count = get_retrieved_file_count(response)
    used_fallback = FALLBACK_PHRASE in response.output_text

    if used_fallback:
        return "acknowledged_gap"
    if not searched:
        return "no_search_performed"
    if result_count == 0:
        return "search_found_nothing"
    return "answered_with_evidence"

This function assigns each response to one of four categories: the model explicitly acknowledged a gap using the expected fallback phrase, no search was attempted at all, a search ran but found nothing, or the response was answered with retrieved evidence present. Logging this classification alongside every response in a production system turns an invisible failure mode into a measurable metric — you can track what fraction of real user questions fall into search_found_nothing over time, which is a direct, actionable signal that your knowledge base has content gaps worth filling.

You can test this classification logic entirely with fake response objects, without any real API call:

class FakeSearchCall:
    def __init__(self, results):
        self.type = "file_search_call"
        self.results = results


class FakeResponse:
    def __init__(self, output_text, output):
        self.output_text = output_text
        self.output = output


def test_classify_response_acknowledged_gap():
    response = FakeResponse(
        output_text="I don't have enough information in the knowledge base to answer that.",
        output=[FakeSearchCall(results=[])],
    )
    assert classify_response(response) == "acknowledged_gap"
    print("PASS: fallback phrase is classified as acknowledged_gap")


def test_classify_response_no_search_performed():
    response = FakeResponse(output_text="Paris is the capital of France.", output=[])
    assert classify_response(response) == "no_search_performed"
    print("PASS: absence of any search call is classified as no_search_performed")


def test_classify_response_search_found_nothing():
    response = FakeResponse(
        output_text="Based on general practice, notice periods are typically 30 days.",
        output=[FakeSearchCall(results=[])],
    )
    assert classify_response(response) == "search_found_nothing"
    print("PASS: an empty result set is classified as search_found_nothing")


def test_classify_response_answered_with_evidence():
    response = FakeResponse(
        output_text="Per the policy document, the notice period is 45 days.",
        output=[FakeSearchCall(results=["chunk_1", "chunk_2"])],
    )
    assert classify_response(response) == "answered_with_evidence"
    print("PASS: non-empty results are classified as answered_with_evidence")


test_classify_response_acknowledged_gap()
test_classify_response_no_search_performed()
test_classify_response_search_found_nothing()
test_classify_response_answered_with_evidence()

Each test constructs a minimal fake response representing one of the four scenarios and asserts classify_response returns the expected category. Notice the third test in particular: it represents the dangerous case where the model answered fluently ("typically 30 days") despite an empty result set — exactly the failure this whole lesson is about — and confirms the classifier correctly flags it as search_found_nothing rather than letting it pass as a normal answer. Having this test suite means you can safely refine the classification logic later without accidentally breaking detection of this specific case.

Escalation Paths for Detected Gaps

Detecting a gap is only useful if something happens as a result. Reasonable responses, depending on your application, include:

• Surfacing a visible "I couldn't find this in the knowledge base" message to the user, rather than a hidden log entry only you see.
• Offering to hand off to a human (a support ticket, a "contact us" prompt) specifically when classify_response returns search_found_nothing or no_search_performed.
• Falling back to web search for questions that are legitimately answerable from current public information but not from your private documents — the subject of Lesson 9.
• Aggregating search_found_nothing classifications over time into a report of candidate topics to add to the knowledge base, closing the loop back into the document preparation work from Lesson 7.

Common Mistakes

Relying solely on prompt instructions to prevent ungrounded answers, without any programmatic verification, which leaves you unable to detect or measure how often the model still answers from general knowledge despite being told not to.

Treating "the model produced text" as equivalent to "the model found evidence," conflating fluency with grounding — always check whether file_search actually ran and returned results, not just whether a coherent answer came back.

Not distinguishing between the three failure modes (no documents exist, documents exist but weren't retrieved, documents were retrieved but ignored), which leads to fixing the wrong part of the system — adding documents when the real problem is chunking, for instance.

Best Practices

Prescribe an exact, detectable fallback phrase in your grounding instructions, so your application code can programmatically recognize when the model is acknowledging a gap rather than parsing free-form uncertainty language.

Classify and log every response's evidence status, not just its final text, so that gaps in your knowledge base become a measurable, trackable metric rather than something only noticed when a user complains.

Build an explicit escalation path for detected gaps — a visible message, a human handoff, or a fallback to web search — rather than letting a detected failure mode dead-end silently in a log file no one reviews.