What "Unsupported Claim" Actually Means Here

An unsupported claim, in the context of a web-search-enabled response, is a statement the model makes that is not actually backed by anything it retrieved during search — even though the response as a whole looks grounded because a search happened and citations are attached elsewhere in the answer. This is a subtler and more common problem than an outright fabrication with zero search involved. The model performs a real search, finds real sources, cites some of them correctly, and then, somewhere in the same answer, adds an additional detail, elaboration, or generalization that came from its own training-derived knowledge rather than from anything it just retrieved.

This matters because a reader has no way to distinguish a well-grounded sentence from an ungrounded one sitting right next to it in the same paragraph, unless the application is specifically designed to make that distinction visible. The earlier lessons in this unit — citations, structured fields for sources, confidence heuristics — all help with whether a source exists at all. This lesson is about the narrower and harder problem of making sure each individual claim in the output is actually tied to something retrieved, not just generally "in the neighborhood" of a search that happened.

Why This Happens: The Model Fills Gaps by Default

Language models are trained to produce fluent, complete-sounding text. When a search result is partial — it answers part of the question but leaves a gap — the model's strong default behavior is to fill that gap using its own general knowledge, producing a smoothly complete answer rather than an answer with a visible hole in it. This is usually a desirable property in ordinary conversation. It becomes a liability specifically when your application's value proposition depends on every claim being traceable to a real, current source.

Consider a search for "what is the current status of a specific pending piece of legislation." A search might return a source describing the bill's content accurately, but say nothing about its current status, since that source might be older. A model asked to give a complete answer may combine the (correctly retrieved) content description with a (not retrieved, likely stale) guess about status, and present both with the same tone, in the same paragraph, often even attaching the one citation it does have to the entire answer rather than just the part it actually supports.

Strategy One: Explicit Instructions to Distinguish Retrieved Facts from Inference

The most direct mitigation is instructing the model, in the prompt itself, to explicitly separate what it found from what it is inferring or already knew, rather than blending them.

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-5.6-terra",
    tools=[{"type": "web_search"}],
    input=(
        "Search for the current status of a major open-source software project's "
        "next planned major release. In your answer, clearly separate two things: "
        "(1) facts you found directly in your search results, each with its source, "
        "and (2) anything you are inferring, assuming, or recalling from general "
        "knowledge rather than from what you searched. Do not blend these together "
        "in the same sentence. Label the second category explicitly as 'not confirmed "
        "by current search results.'"
    ),
)

print(response.output_text)

The instruction to "not blend these together in the same sentence" is doing real work. Without it, a model can technically comply with "separate facts from inference" while still writing one sentence that mixes both, because sentence-level separation is not automatically implied by paragraph-level separation. Being this explicit feels heavy-handed for casual use, but for an application where unsupported claims are a real risk — medical, legal, financial, or safety-relevant domains — this level of explicit instruction is a reasonable and often necessary cost.

Strategy Two: Structured Output That Forces Per-Claim Grounding

Prompt instructions alone rely on the model choosing to comply every time, which is not a strong guarantee. Combining this with structured outputs, as introduced in Lesson 6, gives you a much firmer mechanism: define a schema where every individual claim must carry its own grounding status, so there is no way for the model to produce output that mixes grounded and ungrounded content without marking the distinction.

from pydantic import BaseModel
from openai import OpenAI


class Claim(BaseModel):
    statement: str
    is_grounded_in_search: bool
    source_url: str | None


class GroundedAnswer(BaseModel):
    claims: list[Claim]


client = OpenAI()

response = client.responses.parse(
    model="gpt-5.6-terra",
    tools=[{"type": "web_search"}],
    input=(
        "Search for the current status of a major open-source software project's "
        "next planned major release. Break your answer into individual claims. "
        "For each claim, set is_grounded_in_search to true only if you found that "
        "specific claim in your search results, and include the source_url in that "
        "case. If a claim is your own inference, general knowledge, or assumption "
        "rather than something you found in search results, set is_grounded_in_search "
        "to false and leave source_url empty."
    ),
    text_format=GroundedAnswer,
)

answer: GroundedAnswer = response.output_parsed

for claim in answer.claims:
    tag = "GROUNDED" if claim.is_grounded_in_search else "UNVERIFIED"
    print(f"[{tag}] {claim.statement}")
    if claim.source_url:
        print(f"    source: {claim.source_url}")

This schema makes the grounding status a mandatory, per-claim field rather than an optional nuance buried in prose. is_grounded_in_search is a plain boolean the model must set for every single claim it produces — there is no schema-valid way to produce a claim without declaring which category it falls into. source_url is typed as str | None specifically because an ungrounded claim legitimately has no source to report, and forcing a non-optional field here would either produce an empty string (ambiguous — does empty mean "no source" or "the model failed to fill this in"?) or force the model to fabricate a placeholder URL, which is worse than having no URL at all.

This does not make ungrounded claims disappear — the model can still produce them, and it can still mislabel one if it is genuinely confused about its own reasoning process. What it does is turn an invisible problem into a visible, filterable one: your application can now trivially filter answer.claims down to only is_grounded_in_search=True entries before displaying anything in contexts where unverified content is unacceptable, which was not possible with a single undifferentiated paragraph of text.

Filtering to Only Grounded Claims

def grounded_only(claims: list[Claim]) -> list[Claim]:
    return [c for c in claims if c.is_grounded_in_search and c.source_url]


def test_grounded_only_excludes_unverified_and_sourceless_claims():
    claims = [
        Claim(statement="A", is_grounded_in_search=True, source_url="https://example.com/a"),
        Claim(statement="B", is_grounded_in_search=False, source_url=None),
        Claim(statement="C", is_grounded_in_search=True, source_url=None),
    ]

    result = grounded_only(claims)

    assert len(result) == 1, f"expected only claim A to pass, got {[c.statement for c in result]}"
    assert result[0].statement == "A"

    print("PASS: grounded_only keeps only claims marked grounded AND carrying a source URL")


test_grounded_only_excludes_unverified_and_sourceless_claims()

Note that grounded_only checks both is_grounded_in_search and the presence of source_url, not just the boolean flag alone. This is a deliberate defensive choice: claim C in the test has is_grounded_in_search=True but no source_url, representing a case where the model may have mislabeled a claim as grounded without actually attaching a real source to it. Requiring both conditions is a small extra safeguard against exactly the kind of model inconsistency that this entire lesson is about — the model's own self-reported labels are a strong signal, but not an infallible one, and a defensive application checks the underlying data, not just the label.

Combining This With the Confidence Heuristic from Lesson 7

The per-claim grounding pattern here composes naturally with the confidence scoring introduced in the previous lesson. Rather than scoring an entire response as one unit, you can compute the proportion of claims that are actually grounded, giving a finer-grained signal than a single citation count for the whole answer.

def grounded_ratio(claims: list[Claim]) -> float:
    if not claims:
        return 0.0
    grounded = grounded_only(claims)
    return len(grounded) / len(claims)


def test_grounded_ratio():
    claims = [
        Claim(statement="A", is_grounded_in_search=True, source_url="https://example.com/a"),
        Claim(statement="B", is_grounded_in_search=False, source_url=None),
    ]
    assert grounded_ratio(claims) == 0.5
    assert grounded_ratio([]) == 0.0

    print("PASS: grounded_ratio computes the fraction of claims that are genuinely sourced")


test_grounded_ratio()

A response where grounded_ratio comes back low — say, under half the claims are actually sourced — is a strong signal that the model leaned heavily on its own inference for this particular answer, regardless of how confident the writing sounds. An application can use this threshold to decide whether to show the full answer, show only the grounded claims, or ask the user to rephrase the question to something more search-friendly.

Common Mistakes

Relying only on prompt wording to enforce claim-level separation, which causes inconsistent compliance, since the model can still blend grounded and ungrounded content in a single sentence unless a schema structurally prevents it. Prompt instructions help, but pairing them with a structured, per-claim schema (as shown here) is significantly more reliable.

Treating the model's self-reported is_grounded_in_search flag as infallible, which causes occasional mislabeled claims to slip through as if they were verified. Add a defensive check, such as also requiring a non-empty source_url, rather than trusting the boolean flag in isolation.

Applying this level of rigor uniformly to every feature, which causes unnecessary complexity and latency for low-stakes use cases where a blended, natural-sounding paragraph is perfectly appropriate. Reserve the full per-claim grounding pattern for applications where unsupported claims carry real consequences.

Best Practices

Use a per-claim structured schema with an explicit grounding flag for any application where distinguishing retrieved fact from model inference materially matters, rather than relying on prose alone.

Compute a grounded ratio across claims, not just a whole-response confidence label, to get a finer signal about how much of a given answer is actually backed by current search results.

Design your prompt and schema together. The prompt should tell the model exactly what "grounded" means for your use case (found directly in this search, versus recalled or inferred), and the schema should make that distinction a mandatory field rather than an optional nuance the model might omit.