Why Structured Output Needs Its Own Tests

When the OpenAI SDK is used with a Pydantic model to enforce structured output (via client.responses.parse(...) or a JSON-schema-constrained response_format), your application code typically stops working with raw text and starts working with typed Python objects. That shift changes what needs testing. It is no longer enough to check that a string exists — you need to verify that:

1. Your code correctly defines the schema you intend the model to fill in.
2. Your code correctly extracts and validates a parsed object once the API returns one.
3. Your code handles the case where parsing fails or the API returns something that does not match the schema.

None of this requires calling the real model. It is entirely deterministic Python logic — a schema is a class, and validation is a well-defined operation — which makes it a perfect fit for ordinary unit tests, distinct from an evaluation of whether the model tends to fill the schema in correctly (an evaluation-layer concern, covered later in this unit).

Defining the Schema

A Pydantic model doubles as both the schema you hand to the SDK and the validation logic your tests exercise directly, with no network call involved.

from pydantic import BaseModel, Field


class InvoiceLineItem(BaseModel):
    description: str
    quantity: int = Field(gt=0)
    unit_price: float = Field(gt=0)


class ExtractedInvoice(BaseModel):
    vendor_name: str
    invoice_number: str
    line_items: list[InvoiceLineItem]
    total_amount: float = Field(gt=0)

Field(gt=0) declares a validation constraint — the value must be greater than zero — directly in the schema. This matters because it means invalid data (a negative quantity, a zero total) is rejected by Pydantic itself at construction time, before your application logic ever sees it. Your job in testing is to confirm that this rejection actually happens for the inputs you expect to be invalid, and that valid inputs are accepted without modification.

Testing That Valid Data Parses Successfully

def test_extracted_invoice_accepts_valid_data():
    data = {
        "vendor_name": "Acme Supplies",
        "invoice_number": "INV-2031",
        "line_items": [
            {"description": "Widgets", "quantity": 10, "unit_price": 2.5},
            {"description": "Gadgets", "quantity": 3, "unit_price": 19.99},
        ],
        "total_amount": 84.97,
    }

    invoice = ExtractedInvoice.model_validate(data)

    assert invoice.vendor_name == "Acme Supplies"
    assert len(invoice.line_items) == 2
    assert invoice.line_items[0].quantity == 10
    print("PASS: ExtractedInvoice accepts well-formed data")

ExtractedInvoice.model_validate(data) is the same validation path the SDK's structured-output parsing uses internally: it takes a plain dictionary (which is what JSON deserializes into) and either returns a fully-typed, validated instance or raises pydantic.ValidationError. Testing this path directly, with dictionaries you construct by hand, lets you check your schema's behavior across many realistic and edge-case inputs in milliseconds, without waiting on or paying for a real model call.

Testing That Invalid Data Is Rejected

Equally important is confirming the schema actually rejects data it should reject — a schema with a bug in its constraints can silently accept bad data, defeating the purpose of using structured output at all.

import pytest
from pydantic import ValidationError


def test_extracted_invoice_rejects_negative_quantity():
    data = {
        "vendor_name": "Acme Supplies",
        "invoice_number": "INV-2031",
        "line_items": [
            {"description": "Widgets", "quantity": -5, "unit_price": 2.5},
        ],
        "total_amount": -12.5,
    }

    with pytest.raises(ValidationError):
        ExtractedInvoice.model_validate(data)

    print("PASS: ExtractedInvoice rejects a negative line-item quantity")


def test_extracted_invoice_rejects_missing_required_field():
    data = {
        "vendor_name": "Acme Supplies",
        # "invoice_number" is missing
        "line_items": [],
        "total_amount": 10.0,
    }

    with pytest.raises(ValidationError):
        ExtractedInvoice.model_validate(data)

    print("PASS: ExtractedInvoice rejects data missing invoice_number")

pytest.raises(ValidationError) is a context manager that turns "this code must raise this exception" into an assertion: the test fails if the block completes without raising ValidationError, and it fails if a different exception type is raised instead, which keeps the test precise about what failure mode it is checking for. Writing both a "valid data is accepted" test and a "invalid data is rejected" test for each meaningful constraint is what actually proves the schema behaves as intended — testing only the happy path leaves broken constraints (for example, an accidentally-removed gt=0) completely invisible.

Testing Code That Consumes a Parsed Response

The schema itself is only half the picture. The application code that receives a parsed response from the SDK and does something with it also needs coverage — and this is where the fakes and mocks from earlier lessons combine with Pydantic validation.

class FakeParsedResponse:
    def __init__(self, parsed_obj):
        self.output_parsed = parsed_obj


def extract_invoice(client, document_text: str) -> ExtractedInvoice:
    response = client.responses.parse(
        model="gpt-5.6-terra",
        input=f"Extract invoice details from:\n\n{document_text}",
        text_format=ExtractedInvoice,
    )
    return response.output_parsed


def test_extract_invoice_returns_typed_object():
    parsed = ExtractedInvoice(
        vendor_name="Beta Corp",
        invoice_number="B-1002",
        line_items=[
            InvoiceLineItem(description="Service fee", quantity=1, unit_price=500.0)
        ],
        total_amount=500.0,
    )

    class FakeClient:
        class responses:
            @staticmethod
            def parse(**kwargs):
                return FakeParsedResponse(parsed)

    result = extract_invoice(FakeClient(), "Some raw invoice text.")

    assert isinstance(result, ExtractedInvoice)
    assert result.total_amount == 500.0
    print("PASS: extract_invoice returns a validated ExtractedInvoice instance")

This test never calls the real model and never even calls model_validate directly — it constructs a valid ExtractedInvoice instance up front and checks that extract_invoice correctly plumbs it through from response.output_parsed to its return value. This is a different concern from the schema tests above: those tests check the schema's validation rules; this test checks that your function correctly retrieves and returns the parsed object without corrupting or misreading it (for example, accidentally returning response instead of response.output_parsed).

Handling the Refusal and Malformed-Output Cases

Structured output parsing can fail in ways your code must handle gracefully: the model might refuse to answer, or (particularly when not using strict schema enforcement) return JSON that does not match the schema. Testing these paths means testing your error-handling code, not the model's behavior.

def extract_invoice_safe(client, document_text: str) -> ExtractedInvoice | None:
    response = client.responses.parse(
        model="gpt-5.6-terra",
        input=f"Extract invoice details from:\n\n{document_text}",
        text_format=ExtractedInvoice,
    )
    if getattr(response, "refusal", None):
        return None
    return response.output_parsed


class FakeRefusalResponse:
    def __init__(self):
        self.output_parsed = None
        self.refusal = "The document does not appear to be an invoice."


def test_extract_invoice_safe_handles_refusal():
    class FakeClient:
        class responses:
            @staticmethod
            def parse(**kwargs):
                return FakeRefusalResponse()

    result = extract_invoice_safe(FakeClient(), "Not an invoice at all.")

    assert result is None
    print("PASS: extract_invoice_safe returns None on a model refusal")

getattr(response, "refusal", None) is a defensive read that avoids an AttributeError if the response object does not carry a refusal field at all in a given SDK version — a small but real detail worth testing explicitly, since a change in how refusals are represented is exactly the kind of drift that a unit test using a deliberately-shaped fake will surface immediately, while a happy-path-only test suite would not.

Note: Field names such as output_parsed and refusal reflect this course's conventions for the Responses API; verify exact attribute names against the SDK version you have installed, since structured-output APIs have evolved across releases.

Common Mistakes

• Testing only the happy path of a schema. A constraint like Field(gt=0) that has been accidentally deleted in a refactor will not be caught unless a test specifically asserts that an invalid value (zero or negative) is rejected.
• Conflating "the model filled the schema in well" with "the schema itself is correct." Whether the model reliably produces sensible line_items for real invoices is an evaluation question; whether ExtractedInvoice rejects a negative quantity is a unit-testing question — mixing them into one test makes failures ambiguous.
• Not testing the refusal or malformed-output path. Code that assumes response.output_parsed is always a valid object will crash in production the first time the model declines to answer or returns an empty result, a case that is trivial to simulate with a fake but easy to forget.

Best Practices

• Write a rejection test for every meaningful field constraint, not just an acceptance test for the happy path, so a broken constraint is caught immediately.
• Test the extraction/consumption code and the schema's validation rules separately — one confirms your function correctly reads the parsed object, the other confirms the schema enforces the right rules.
• Explicitly test the refusal and malformed-output branches of any function that consumes structured output, using a fake response shaped exactly like the failure case you are guarding against.