The Open Web Is Not a Reliable Database

Every technique so far in this unit has assumed, implicitly, that a search will return something useful. In practice, the open web is a mixture of authoritative primary sources, reasonable secondary reporting, outdated pages that were never updated after the facts changed, speculative or opinion content presented as fact, and outright low-quality or spam content optimized to rank in search rather than to be accurate. A grounded answer is only as good as what it is grounded in — and unlike a curated internal database, you do not control what exists on the web your search tool draws from.

This creates two distinct failure modes an application needs to handle deliberately, rather than hoping the model sorts them out on its own:

1. Conflicting sources: two or more reasonably credible sources disagree, for example about a statistic, a date, or a current status, often because they were published at different times or use different methodologies.
2. Low-quality sources: a source is outdated, unreliable, or simply wrong, and there may be no visible conflict at all if it is the only source found — the danger here is a confident, single-source answer with no signal that the source itself is weak.

Handling these well is what separates a genuinely trustworthy grounded application from one that merely looks grounded because it has citations attached.

Instructing the Model to Surface Disagreement Rather Than Resolve It Silently

The default failure mode, if you say nothing about it, is that a model tends to pick one plausible answer and present it confidently, even when its own search results disagreed. This happens because generating a single, fluent, confident answer is what these models are optimized to do by default — surfacing uncertainty or disagreement is something you need to explicitly ask for, not something that happens automatically.

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-5.6-terra",
    tools=[{"type": "web_search"}],
    input=(
        "Search for the current estimated number of active users of a major "
        "social media platform of your choice. If different sources give "
        "different figures, do not simply pick one — explicitly state each "
        "figure you found, which source it came from, and how recent each "
        "source appears to be. If the sources roughly agree, say so as well."
    ),
)

print(response.output_text)

The key instruction here is explicit: "do not simply pick one — explicitly state each figure you found." Without this, the model has no strong incentive to slow down and enumerate disagreement rather than quietly averaging or picking whichever source it processed most recently. This connects directly to the prompt design principle from Lesson 3 — what the model does with search results is heavily shaped by what you ask it to do with them, not just by whether search happened at all.

This is also a case where structured outputs, from Lesson 6, genuinely help. A free-text answer can mention disagreement in a way that is easy for a human to skim past. A schema that has an explicit conflicting boolean field and a list of alternative_figures forces the model to make a decision about whether disagreement exists, and forces your application to handle that case deliberately in code, rather than leaving it buried in prose.

from pydantic import BaseModel
from openai import OpenAI


class SourcedFigure(BaseModel):
    value_description: str
    source_url: str
    apparent_recency: str


class FigureReport(BaseModel):
    sources_agree: bool
    figures: list[SourcedFigure]
    summary: str


client = OpenAI()

response = client.responses.parse(
    model="gpt-5.6-terra",
    tools=[{"type": "web_search"}],
    input=(
        "Search for the current estimated number of active users of a major "
        "social media platform of your choice. Report every distinct figure "
        "you find as a separate entry, along with its source and how recent "
        "it appears to be. Set sources_agree to false if the figures meaningfully differ."
    ),
    text_format=FigureReport,
)

report: FigureReport = response.output_parsed

if not report.sources_agree:
    print("Sources disagree. Reported figures:")
    for figure in report.figures:
        print(f"  - {figure.value_description} (source: {figure.source_url}, {figure.apparent_recency})")
else:
    print("Sources agree:", report.summary)

Here, sources_agree is a plain boolean the application can branch on directly with an if statement, rather than needing to parse free text for hedging language like "however" or "on the other hand" to detect disagreement. This is a strong example of why structured outputs pair so well with search-grounded generation specifically: the messiest, most application-relevant judgment — is this settled or not — becomes a typed field you can act on, instead of a nuance buried in a paragraph.

Detecting Low-Quality or Unsupported Answers Programmatically

Conflicting sources are at least visible — the model can say "these disagree." A single low-quality source, with nothing to contrast it against, is harder to catch, because nothing about the response looks unusual. It reads exactly like a well-supported answer.

A few practical, application-level signals help here, none of which is perfect on its own but which are useful in combination:

• No citation at all despite search being enabled. If response.output contains a search step but the resulting message has no annotations, that is a signal the model may have searched but then answered from its own knowledge anyway, or found nothing citable. Treat this as lower confidence.
• Very few distinct sources for a claim that would normally have many. A well-established, widely reported fact should easily surface multiple sources. If a search for something significant returns only one obscure source, that is worth flagging rather than trusting outright.
• Domain filtering from Lesson 3, applied proactively, is itself a low-quality-source mitigation — restricting the initial pool of pages the tool can draw from is often more reliable than trying to judge quality after the fact from a single response.

def assess_confidence(citation_count: int, sources_agree: bool | None) -> str:
    """A simple heuristic combining source count and agreement into a confidence label."""
    if citation_count == 0:
        return "low"
    if sources_agree is False:
        return "medium"
    if citation_count == 1:
        return "medium"
    return "high"


def test_assess_confidence():
    assert assess_confidence(citation_count=0, sources_agree=None) == "low"
    assert assess_confidence(citation_count=1, sources_agree=True) == "medium"
    assert assess_confidence(citation_count=3, sources_agree=False) == "medium"
    assert assess_confidence(citation_count=3, sources_agree=True) == "high"

    print("PASS: assess_confidence produces expected labels across citation count and agreement combinations")


test_assess_confidence()

assess_confidence is deliberately simple — a small, explicit heuristic rather than an attempt at a comprehensive scoring model. Zero citations is always treated as low confidence, since an application that promises grounded answers should not present an ungrounded one as if it were equally reliable. Disagreement between sources or having only a single source both cap confidence at "medium," since neither situation warrants full confidence even though an answer was produced. Only multiple sources that agree earns "high." This kind of heuristic is meant to drive a UI decision — for example, showing a "verify this" badge on medium or low confidence answers — not to be a rigorous statistical measure. Being explicit and simple keeps it easy to reason about, adjust, and test, which matters more for this kind of interpretability-driven logic than squeezing out marginal precision.

Deciding When to Refuse Rather Than Answer

Some applications should be willing to say "I could not find a reliable answer" rather than always producing something. This is a design decision, not a limitation to work around — for high-stakes use cases, a clear non-answer is far better than a low-confidence answer presented with the same tone as a well-supported one.

def format_response_for_user(answer: str, confidence: str) -> str:
    if confidence == "low":
        return (
            "I was not able to find a reliable, well-sourced answer to this. "
            "Please verify independently before relying on this information:\n\n" + answer
        )
    if confidence == "medium":
        return "Note: this answer is based on limited or partially conflicting sources.\n\n" + answer
    return answer


def test_format_response_for_user_adds_warnings_appropriately():
    low = format_response_for_user("Some answer text.", "low")
    medium = format_response_for_user("Some answer text.", "medium")
    high = format_response_for_user("Some answer text.", "high")

    assert low.startswith("I was not able to find")
    assert medium.startswith("Note: this answer is based on limited")
    assert high == "Some answer text."

    print("PASS: format_response_for_user attaches the correct warning per confidence level")


test_format_response_for_user_adds_warnings_appropriately()

This function keeps the confidence assessment and the user-facing presentation as separate, individually testable steps — assess_confidence decides the label, format_response_for_user decides what to show based on that label. Keeping these separate means you can tune the wording of your warnings, or the thresholds for each confidence level, independently of each other, without one change accidentally affecting the other's logic.

Common Mistakes

Trusting a fluent, confident answer as evidence of quality, which happens because a model's writing style does not change based on how good its underlying sources were. Confidence in tone and confidence in content are unrelated; only structured signals like citation count and explicit agreement checks tell you anything about the latter.

Asking the model to "just give me the answer" for topics likely to have conflicting current data, which causes the model to silently pick one source and discard the disagreement, because that produces a shorter, more satisfying-sounding response. Explicitly instruct the model to surface disagreement, as shown in this lesson, whenever the topic is one where sources are likely to differ.

Treating zero citations as equivalent to a citation-backed answer, which causes an ungrounded fallback response (the model answering from its own training data because a search failed or found nothing) to be presented with the same confidence as a well-sourced one. Always check whether citations exist before deciding how much to trust a search-enabled response.

Best Practices

Use a structured field like sources_agree rather than relying on free-text hedging to detect and act on source disagreement, since a boolean is something your application can reliably branch on.

Build a simple, explicit confidence heuristic based on citation count and agreement, and use it to drive real UI or logic decisions — such as showing a warning or declining to answer — rather than treating every grounded response as equally trustworthy.

Design your application to be willing to say "I don't have a reliable answer" for high-stakes questions, rather than always forcing a confident-sounding response regardless of source quality.