Why This Category of Testing Is Different

Unit 13 covered evaluating and improving AI applications in general — building test cases, scoring outputs, and iterating based on measured quality. Freshness-sensitive features, the kind this entire unit has been building, introduce a specific complication that general-purpose evals do not fully address: the correct answer to a test case can change over time, through no fault of your code at all.

If you write a test that asserts "the answer must say the current population of a city is exactly X," that test is only true on the day you wrote it. A week later, if the real figure changed, or if the web search tool happens to surface a differently-worded but still-correct source, your test starts failing for reasons that have nothing to do with a bug in your application. This is a fundamentally different testing problem than the deterministic, fake-object-based tests used throughout this unit for things like citation extraction or claim filtering — those test pure application logic with fixed inputs, and freshness-sensitive answers, by definition, do not have fixed correct outputs.

This lesson is about the specific practices that make freshness-sensitive behavior testable at all: separating what you can test deterministically from what you must test with looser, structural assertions, and being deliberate about the difference.

What You Can and Cannot Test Deterministically

The first step is drawing a clear line between two categories of behavior in a search-grounded feature:

Deterministic, testable with fixed assertions: anything that is your own code's logic operating on a fixed input — citation extraction (Lesson 4), claim filtering (Lesson 8), confidence scoring (Lesson 7), prompt construction. All of this was already shown using fake response objects and plain assert statements earlier in this unit, precisely because none of it depends on what the live web currently contains.

Non-deterministic, requiring structural or property-based checks: the actual content of a live search-backed answer. You cannot assert "the answer says X" when X might legitimately change. What you can assert is that the response has the right shape and properties — for example, that it includes at least one citation, that it correctly reports the reference date you gave it, or that a schema-validated response actually parses.

Confusing these two categories is the single most common mistake in testing this kind of feature: writing a brittle test that pins down live, changeable content, which then fails constantly for reasons unrelated to your code, eventually leading a team to disable or ignore that test entirely — which defeats the purpose of having tests in the first place.

Testing the Deterministic Layer Thoroughly

Every function built in this unit that does not itself call the API — get_citations, assess_confidence, grounded_only, validate_price_report, the ResearchAssistant's prompt construction — belongs in this layer, and should have thorough, fixed-assertion tests, exactly as shown in earlier lessons. This is worth restating clearly because it is the majority of the testing effort that should go into a freshness-sensitive feature: most of what can actually go wrong in your code is in the deterministic layer, not in "did the web return the right fact today."

from datetime import date, timedelta


def build_freshness_prompt(topic: str, reference_date: date, max_age_days: int) -> str:
    cutoff = reference_date - timedelta(days=max_age_days)
    return (
        f"Today's date is {reference_date.isoformat()}. "
        f"Search for current information about: {topic}. "
        f"Only use sources you believe were published on or after {cutoff.isoformat()}. "
        "If you cannot find a source that recent, say so explicitly rather than "
        "using an older source."
    )


def test_build_freshness_prompt_computes_correct_cutoff():
    prompt = build_freshness_prompt(
        topic="a hypothetical product launch",
        reference_date=date(2026, 9, 14),
        max_age_days=14,
    )

    assert "Today's date is 2026-09-14" in prompt
    assert "on or after 2026-08-31" in prompt
    assert "a hypothetical product launch" in prompt

    print("PASS: build_freshness_prompt correctly computes the cutoff date and embeds all inputs")


test_build_freshness_prompt_computes_correct_cutoff()

This test locks down the date arithmetic (reference_date - timedelta(days=max_age_days)) completely deterministically, using a fixed date(2026, 9, 14) rather than date.today(). This is an important detail: passing a fixed date into the function under test, rather than letting it call date.today() internally, is exactly what makes this test reproducible regardless of what day it actually runs. If build_freshness_prompt had called date.today() internally instead of receiving reference_date as a parameter, this test would need to compute the expected cutoff relative to "whenever this test happens to run," which is a needless complication. Designing functions to receive time-related values as parameters, rather than reaching for the current time internally, is a general testability principle that matters even more for freshness-sensitive code specifically.

Testing the Non-Deterministic Layer with Structural Assertions

For the parts of the system that do call the live API and depend on genuinely current web content, replace "does it say the exact right thing" with "does it have the right properties." This is sometimes called a property-based or structural check, and it is the right tool when the exact expected value is inherently unstable.

from openai import OpenAI

client = OpenAI()


def check_response_structure(response) -> list[str]:
    """Structural checks for a search-grounded response. Returns a list of problems found."""
    problems = []

    has_search_call = any(item.type == "web_search_call" for item in response.output)
    if not has_search_call:
        problems.append("No web_search_call item found in response.output.")

    has_message = any(item.type == "message" for item in response.output)
    if not has_message:
        problems.append("No message item found in response.output.")

    has_citation = False
    for item in response.output:
        if item.type != "message":
            continue
        for content_block in item.content:
            if getattr(content_block, "annotations", None):
                has_citation = True

    if not has_citation:
        problems.append("No citations found despite web search being requested.")

    if not response.output_text.strip():
        problems.append("output_text is empty.")

    return problems


response = client.responses.create(
    model="gpt-5.6-terra",
    tools=[{"type": "web_search"}],
    input="What is the current exchange rate trend between two major world currencies today?",
)

issues = check_response_structure(response)
if issues:
    print("Structural check failed:")
    for issue in issues:
        print(" -", issue)
else:
    print("PASS: response has expected structure (search occurred, message present, citation present, non-empty text)")

check_response_structure deliberately never checks what the answer says — only that a search item appears, that a message was produced, that at least one citation exists, and that the final text is non-empty. These are exactly the kind of properties that should hold true regardless of what the current exchange rate actually is on any given day, which makes them stable, meaningful things to assert against a live call, unlike a hardcoded expected value would be.

Note: The exact item type string used to detect a search step (web_search_call here) is a version-sensitive API detail, matching the same caveat from Lesson 2. Confirm this against your SDK's current documentation, since a structural check like this one silently stops working correctly if the underlying type name changes and nothing alerts you to it.

Running Live Checks Sparingly and Deliberately

Because structural checks like the one above still make a real API call, they are slower, cost money, and can occasionally fail for reasons outside your control — the web search backend having a transient issue, for instance. The general pattern used across this course, consistent with Unit 13's approach to evaluation, is to keep these as a small, separate suite from your fast, free, deterministic unit tests, run less frequently (for example, in a scheduled check rather than on every code change), and treated as a signal to investigate rather than an automatic failure that blocks work.

def summarize_structural_check(problems: list[str]) -> str:
    if not problems:
        return "OK: search-grounded response passed all structural checks."
    return "NEEDS REVIEW: " + "; ".join(problems)


def test_summarize_structural_check_formats_both_cases():
    assert summarize_structural_check([]) == "OK: search-grounded response passed all structural checks."
    assert summarize_structural_check(["No citations found."]).startswith("NEEDS REVIEW:")

    print("PASS: summarize_structural_check produces a clear pass/review message for both cases")


test_summarize_structural_check_formats_both_cases()

Separating "run the structural check" (which needs the live API) from "summarize the result" (pure logic, fully testable with fixed lists) again reflects the same principle from this whole lesson: push as much logic as possible into the deterministic, easily-testable layer, and keep the live-API-dependent surface as small and simple as it can be.

A Checklist for Testing a Freshness-Sensitive Feature

The last row is worth being honest about: verifying that the actual current fact reported by the model is correct, on any given day, is not something an automated test suite can meaningfully assert without itself becoming another unreliable, unverified source of truth. This is where human review, periodic spot-checks, or comparison against a small number of trusted reference sources fits in — a complement to automated testing, not a replacement for the structural and deterministic checks that automation handles well.

Common Mistakes

Writing tests that hardcode an expected fact that can change, which causes the test to fail on a schedule unrelated to actual bugs, eventually training the team to ignore failing tests altogether. Replace fact-specific assertions with structural ones for anything backed by live search.

Calling date.today() inside functions you intend to test, which causes tests to need to reproduce "whatever today is" instead of asserting against a fixed, known value. Accept the reference date as a parameter, as shown in build_freshness_prompt, so tests can pass in a fixed date.

Running live, API-calling structural checks as part of every fast unit test run, which causes the fast test suite to become slow, costly, and occasionally flaky due to network issues unrelated to code correctness. Keep deterministic and live-API tests in clearly separated suites, run at different frequencies.

Best Practices

Push as much logic as possible into pure, deterministic functions — prompt building, citation handling, confidence scoring — specifically because doing so maximizes the portion of your freshness-sensitive feature that can be tested quickly, cheaply, and reliably with fixed assertions.

When you must test a live, search-backed call, assert on structure and properties, not exact content — that a search occurred, that a citation exists, that the output is non-empty — since these properties remain meaningful regardless of what the current facts happen to be on any given day.

Treat periodic human spot-checks of actual answer content as a necessary complement to automated testing for freshness-sensitive features, rather than trying to force full content verification into an automated suite where it does not belong.