Why "It Ran Real Code" Is Not the Same as "It's Correct"

Lesson 1 established the central reason code interpreter exists: real executed Python code produces exact, reproducible values instead of the model's statistical guess. It is tempting to treat that as the end of the accuracy story — the code ran, therefore the number is right. It is not that simple. The execution is exact, but the logic the model chose to execute is still something the model decided, and that decision can be wrong in ways that produce a perfectly-computed, perfectly-confident, and perfectly-incorrect answer. A model can write code that filters the wrong rows, joins tables on the wrong key, computes a mean instead of a weighted mean, or silently drops rows with missing values when the analysis called for imputing them instead. The arithmetic in each case is flawless; the analysis is wrong.

This is the gap this lesson addresses: treating code interpreter's output the way you would treat a junior analyst's report — generally competent, but worth checking against known facts before it drives a business decision, especially in a system running unattended.

Technique 1: Ask the Model to Show Its Work

The single highest-leverage validation technique costs nothing extra: request that the generated code print its intermediate steps, not just the final number.

response = client.responses.create(
    model="gpt-5.6-terra",
    tools=[{
        "type": "code_interpreter",
        "container": {"type": "auto", "file_ids": [uploaded.id]},
    }],
    input=(
        "Compute the average order value in orders.csv. Print the row "
        "count before and after any filtering, the sum of the amount "
        "column, and the final average, each on its own line, before "
        "giving your final answer."
    ),
)

for item in response.output:
    if item.type == "code_interpreter_call":
        print(item.code)

Printing intermediate values converts a single opaque number into an inspectable chain of reasoning you can check by eye: if the row count after filtering is suspiciously low, or the sum looks implausible for the sum of the actual column, that is visible immediately, before you ever look at the final average. This is directly analogous to asking a person to show their work on a math problem — not because you doubt arithmetic, but because most errors live in the steps, not the final operation.

Technique 2: Independent Recomputation

For any result that will drive an automated decision (triggering an alert, feeding a downstream report, appearing in a customer-facing dashboard), the most reliable check is computing the same value yourself, independently, using code you wrote and trust — and comparing the two.

import pandas as pd


def independently_verify_average_order_value(csv_path: str, models_claimed_value: float, tolerance: float = 0.01) -> bool:
    df = pd.read_csv(csv_path)
    actual_average = df["amount"].mean()
    difference = abs(actual_average - models_claimed_value)
    is_valid = difference <= tolerance * actual_average
    if not is_valid:
        print(
            f"Validation failed: model claimed {models_claimed_value}, "
            f"independent computation found {actual_average} "
            f"(difference {difference})"
        )
    return is_valid


def test_independently_verify_average_order_value():
    import tempfile
    import os

    csv_content = "amount\n100\n200\n300\n"
    fd, file_path = tempfile.mkstemp(suffix=".csv")
    with os.fdopen(fd, "w") as f:
        f.write(csv_content)

    try:
        assert independently_verify_average_order_value(file_path, 200.0) is True
        assert independently_verify_average_order_value(file_path, 50.0) is False
    finally:
        os.remove(file_path)
    print("PASS: independently_verify_average_order_value matches and flags correctly")


test_independently_verify_average_order_value()

This function runs entirely in your own trusted environment — no model call, no code interpreter — using the same raw file the model analyzed. It computes the true average with plain pandas and compares it to whatever value the model reported, allowing for a small floating-point tolerance rather than requiring an exact bitwise match (comparing floating-point numbers for exact equality is unreliable regardless of who computed them, since different computation orders can produce tiny representational differences). Note that the test here writes a small real temporary file rather than using a pure fake object — this is appropriate because the function under test genuinely reads a file from disk, and that behavior is exactly what needs verifying; the important thing preserved from this course's testing pattern is that no live API call happens anywhere in the test.

Independent recomputation is the strongest validation technique available because it does not trust the model's process at any point — only the final claimed value is checked, against ground truth computed by code you wrote and reviewed. It is also the most expensive to build, since it requires you to reimplement, in your own code, whatever computation you are checking — which is only worth doing for values important enough to justify that duplication.

Technique 3: Consistency Checks Across Related Values

When a model reports multiple related numbers in the same analysis, they should be internally consistent with each other even without checking any of them against an external source. This is cheaper than full independent recomputation and catches a surprisingly large share of real errors:

def check_internal_consistency(total_revenue: float, region_revenues: dict[str, float], tolerance: float = 0.01) -> list[str]:
    problems = []
    summed = sum(region_revenues.values())
    if abs(summed - total_revenue) > tolerance * total_revenue:
        problems.append(
            f"Region revenues sum to {summed}, which does not match "
            f"reported total_revenue of {total_revenue}"
        )
    for region, value in region_revenues.items():
        if value < 0:
            problems.append(f"Region '{region}' has a negative revenue value: {value}")
        if value > total_revenue:
            problems.append(f"Region '{region}' revenue ({value}) exceeds total revenue")
    return problems


def test_check_internal_consistency_detects_mismatched_total():
    problems = check_internal_consistency(
        total_revenue=1000.0,
        region_revenues={"East": 300.0, "West": 300.0},
    )
    assert any("does not match" in p for p in problems)
    print("PASS: check_internal_consistency detects a total that doesn't match its parts")


def test_check_internal_consistency_passes_for_consistent_data():
    problems = check_internal_consistency(
        total_revenue=1000.0,
        region_revenues={"East": 400.0, "West": 600.0},
    )
    assert problems == []
    print("PASS: check_internal_consistency finds no issues in consistent data")


test_check_internal_consistency_detects_mismatched_total()
test_check_internal_consistency_passes_for_consistent_data()

This kind of check requires no access to the original dataset at all — it only needs the numbers the model already reported, checked against basic arithmetic and domain rules ("parts sum to the whole," "a percentage of a whole cannot exceed the whole," "revenue cannot be negative"). Because it is cheap to run and needs no extra data access, it is reasonable to run this kind of consistency check on every analysis result in a production pipeline, reserving full independent recomputation (Technique 2) for a smaller set of high-stakes values.

Technique 4: Re-Asking with a Different Approach

For a result you are specifically suspicious of, asking the model to solve the same problem a second time using a deliberately different method is a useful, low-effort cross-check:

response_a = client.responses.create(
    model="gpt-5.6-terra",
    tools=[{"type": "code_interpreter", "container": {"type": "auto", "file_ids": [uploaded.id]}}],
    input="Compute the median order amount in orders.csv using pandas' median() method.",
)

response_b = client.responses.create(
    model="gpt-5.6-terra",
    tools=[{"type": "code_interpreter", "container": {"type": "auto", "file_ids": [uploaded.id]}}],
    input=(
        "Compute the median order amount in orders.csv without using "
        "pandas' built-in median function — sort the values manually and "
        "find the middle element(s) yourself."
    ),
)

print(response_a.output_text)
print(response_b.output_text)

If both approaches agree, that is meaningful evidence the result is correct — two independently written code paths producing the same value rules out a whole category of implementation-specific bugs. If they disagree, you have caught a real problem before it reached a downstream system, and the disagreement itself (which method handled edge cases like even-length lists differently, for instance) often points directly at the bug.

Choosing How Much Validation Is Enough

Validation has a real cost in engineering time and, for some techniques, extra API calls — matching the validation effort to what the result actually drives is the practical way to apply this rather than over-engineering every single computed value in a system.

Common Mistakes

Treating a confident, well-formatted explanation as evidence of correctness. The model's prose explaining its result is generated after the fact and will sound coherent regardless of whether the underlying computation was correct — fluency is not a signal of accuracy.

Only validating the final answer's format, never its value. This is the same trap discussed in Lesson 8 with structured outputs: a syntactically perfect, semantically wrong number passes any check that only inspects shape.

Building independent recomputation for every single value in a system, regardless of stakes. This is expensive to build and maintain, and the effort is better spent concentrated on the smaller number of values that actually drive consequential decisions.

Best Practices

Default to show-your-work prompting on every analysis request — it is free, and it turns every result into something a human (or an automated check) can inspect rather than an opaque final number.

Reserve independent recomputation for values that trigger automated downstream actions, and build it as ordinary, well-tested application code rather than another model call.

Log both the executed code and the final answer for every production analysis request, so that when a validation check does fail, you can immediately see what logic produced the wrong value instead of having to reproduce the failure from scratch.