Two Different Questions About Correctness

When you build an application that calls a large language model through the OpenAI SDK, you are really building two things at once: ordinary application code (parsing arguments, calling the API, handling errors, storing results) and a component whose core behavior — the text or structured data the model produces — is not fully under your control.

Traditional unit testing answers one question: "Does my code do what I told it to do?" You call a function with known inputs, and you assert that the output matches an exact, predictable value. If add(2, 3) does not return 5, the test fails, and it fails the same way every single time you run it.

Evaluation (covered in Unit 13) answers a different question: "Is the model's behavior good enough for real inputs?" You cannot assert that client.responses.create(input="Summarize this article") returns one exact string, because a language model is not a pure function — the same input can legitimately produce different, equally valid outputs. Unit 13 built datasets, graders, and metrics specifically to measure that kind of quality.

This unit is about the piece Unit 13 deliberately set aside: testing the surrounding application code — the parts of your system that are deterministic and do have a single correct answer, even though they sit right next to a non-deterministic model call. Unit 13, Lesson 1 drew this exact line: testing application logic with fakes versus evaluating model behavior with real inputs. This unit takes that distinction and builds an entire testing discipline around the "testing application logic" half.

Why Ordinary Unit Tests Are Still Necessary

It is tempting, once you have an eval pipeline, to think you no longer need conventional tests — after all, the eval will catch the "AI produced garbage" case. But an eval pipeline typically does not catch these bugs:

• A function that constructs the wrong messages array before ever sending it to the model.
• A JSON-parsing bug that raises an unhandled exception when a structured output has an unexpected key order.
• A retry loop that retries forever instead of backing off.
• A tool-dispatch table that calls get_wheather because of a typo, silently doing nothing when the model calls get_weather.
• A regression introduced by a refactor that has nothing to do with prompts or models at all.

These are ordinary software bugs. They deserve ordinary software tests: fast, deterministic, run on every commit, and requiring no API key, network access, or API spend. An eval run, by contrast, is comparatively slow, costs money, and is designed to measure a fuzzy quality signal — it is the wrong tool for catching a typo in a dictionary key.

def build_messages(user_input: str, system_prompt: str) -> list[dict]:
    if not user_input.strip():
        raise ValueError("user_input must not be empty")
    return [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_input},
    ]


def test_build_messages_happy_path():
    messages = build_messages("Hello", "You are a helpful assistant.")
    assert messages[0]["role"] == "system"
    assert messages[1]["content"] == "Hello"
    print("PASS: build_messages returns correctly ordered messages")


def test_build_messages_rejects_empty_input():
    try:
        build_messages("   ", "You are a helpful assistant.")
        raised = False
    except ValueError:
        raised = True
    assert raised, "Expected ValueError for empty input"
    print("PASS: build_messages rejects empty input")

build_messages never touches the network. Its output is fully determined by its input, so a normal assert is exactly the right verification tool. There is no ambiguity to grade — either the dictionary has the right shape or it does not. Running this test costs nothing, takes microseconds, and never depends on OpenAI's servers being reachable. This is precisely the kind of code this unit focuses on: the plumbing around the model call, not the model's judgment.

The Three-Layer Testing Model for AI Applications

A production AI application benefits from thinking about verification in three layers, each with a different tool, a different speed, and a different question it answers.

Unit tests use fakes or mocks so that no real API call happens at all — you are testing your code, not the model. Integration tests make a small number of real (or realistically simulated) calls to confirm your code correctly serializes requests and parses real response objects — you are testing the contract between your code and the SDK, not the quality of the text. Evaluations run a batch of representative inputs through the full pipeline and grade the outputs for quality, consistency, and correctness at the level of meaning, not syntax.

A common mistake is collapsing these layers into one thing — either skipping unit tests because "it's just an AI feature" or trying to make an eval dataset serve as your regression test suite. Each layer catches failures the others cannot. A well-known example: an eval score can look perfectly stable while your tool-call dispatcher silently throws KeyError in production, because the eval was measuring text quality, not exception handling in the surrounding code.

What "Testable" Application Code Looks Like

The reason this unit can meaningfully test SDK-calling code at all is that well-structured code separates decision logic (deterministic, testable) from the API call itself (non-deterministic, mockable). If your code intermixes both — for example, building a prompt, calling the API, and parsing the result all inside one giant function with no seams — you cannot test any piece of it in isolation. You will see this separation pattern used repeatedly, starting in the next lesson.

def extract_ticket_priority(raw_model_output: str) -> str:
    """Pure post-processing logic: no network call, fully testable."""
    normalized = raw_model_output.strip().lower()
    valid_priorities = {"low", "medium", "high", "urgent"}
    if normalized not in valid_priorities:
        raise ValueError(f"Unexpected priority value: {raw_model_output!r}")
    return normalized


def test_extract_ticket_priority_normalizes_case():
    assert extract_ticket_priority("  HIGH \n") == "high"
    print("PASS: extract_ticket_priority normalizes whitespace and case")


def test_extract_ticket_priority_rejects_unknown_values():
    try:
        extract_ticket_priority("critical")
        raised = False
    except ValueError:
        raised = True
    assert raised
    print("PASS: extract_ticket_priority rejects values outside the known set")

extract_ticket_priority never calls the model — it only processes a string the model might have produced. That is exactly what makes it unit-testable: it has no dependency on an external service, so its behavior is fully within your control and fully predictable. Whether the model actually tends to output valid priority values in practice is an evaluation question (Unit 13); whether your parsing code handles both valid and invalid strings correctly is a unit-testing question (this unit).

Note: Throughout this unit, code samples use model="gpt-5.6-terra" as a placeholder model name, consistent with the rest of this course. Substitute the model identifier your account actually has access to.

Common Mistakes

• Treating eval failures and code bugs as the same category of problem. A failing eval might mean the model is genuinely producing worse answers, or it might mean a parsing bug is corrupting otherwise-good output before it reaches the grader. Without a solid unit-test layer underneath, you cannot tell which one you are looking at.
• Skipping unit tests because "the model is nondeterministic anyway." Nondeterminism lives in the model's output, not in your build_messages, extract_ticket_priority, or retry-handling code. That code is exactly as deterministic as any other Python function and deserves exactly the same testing discipline.
• Writing tests that make real API calls by default. A test suite that silently costs money and requires network access every time someone runs pytest will be run less often, defeating the purpose of having fast, cheap tests in the first place.

Best Practices

• Isolate the deterministic parts of your pipeline — message construction, output parsing, validation, routing — into small functions with no hidden dependencies, so each one can be unit tested without touching the network.
• Reserve evaluation datasets and graders for questions about output quality, and reserve unit tests for questions about code correctness; keep both, because neither substitutes for the other.
• Make the default test run fast and free. Real API calls, when needed for integration testing, should be explicit, clearly marked (for example with a pytest marker), and excluded from the test run developers execute dozens of times a day.