Why Free-Text Answers Aren't Enough for Real Pipelines

Every example so far has printed response.output_text — a free-form string. That's fine for a human reading a chat window, but it's a poor fit for code that needs to store a value in a database column, populate a form, or trigger a business rule. If you ask the model for "the total and the date" and the answer comes back as the sentence "The total is $42.50, and the receipt is dated March 3rd," your code now has to parse a sentence to extract a number and a date — fragile work that structured output eliminates entirely.

Unit 6 covered structured outputs in the text-only context: defining a schema for the shape of the response and having the SDK enforce that the model's output matches it exactly. That same mechanism works with image input. This lesson combines the two: an image goes in, and a strictly-shaped object comes out.

Defining a Schema for Image-Derived Data

The most convenient way to define a schema in Python is with Pydantic, and the SDK integrates with it through the responses.parse helper:

from openai import OpenAI
from pydantic import BaseModel
import base64

client = OpenAI()


class ReceiptData(BaseModel):
    merchant_name: str
    total_amount: float
    purchase_date: str
    currency: str


def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")


image_b64 = encode_image("receipts/coffee_shop.jpg")

response = client.responses.parse(
    model="gpt-5.6-terra",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Extract the merchant name, total amount, purchase date, and currency from this receipt.",
                },
                {
                    "type": "input_image",
                    "image_url": f"data:image/jpeg;base64,{image_b64}",
                    "detail": "high",
                },
            ],
        }
    ],
    text_format=ReceiptData,
)

receipt = response.output_parsed
print(receipt.merchant_name, receipt.total_amount, receipt.purchase_date, receipt.currency)

Walking through what changed compared to earlier lessons:

• ReceiptData is a Pydantic model declaring exactly four fields, each with a specific type: two strings, one float, one string used for currency code. This is the schema — it defines the shape the answer must take, not just a description of what you'd like to see.
• client.responses.parse (rather than create) is used together with text_format=ReceiptData. This tells the SDK to constrain the model's output to match the schema and to automatically parse the result into an actual ReceiptData instance for you.
• response.output_parsed gives you that instance directly — a real Python object with real attributes, not a string you have to parse yourself. receipt.total_amount is already a float, ready to use in a calculation, not a string like "$42.50" that still needs cleaning.

Why does this matter more with images than with plain text? Because unconstrained text answers about images are especially prone to including extra commentary — hedges like "it looks like the total might be around $42, though the print is a bit faint" — that make naive string parsing even less reliable than it already is for text-only tasks. Structured output removes that ambiguity: the field is either populated with the model's best value or, with appropriate schema design, marked absent — not buried inside a paragraph of hedging prose.

Handling Fields That Might Not Be Visible

Real images are imperfect. A receipt might be torn, a screenshot might be cropped, and a field your schema expects might genuinely not be present. Declare optional fields explicitly rather than forcing the model to invent a value:

from typing import Optional
from pydantic import BaseModel


class ReceiptData(BaseModel):
    merchant_name: str
    total_amount: Optional[float] = None
    purchase_date: Optional[str] = None
    currency: Optional[str] = None

With Optional[float] = None, the schema tells both the model and your own code that total_amount may legitimately be absent. Your downstream code can then check if receipt.total_amount is None: and branch accordingly — asking the user to confirm the amount manually, for instance — instead of silently trusting a guessed number that happened to satisfy a required field. This is the same anti-hallucination principle from Lesson 4 ("write 'not found' for missing fields"), but expressed as an actual type-level guarantee instead of a string convention you'd otherwise have to parse for yourself.

Note: The exact behavior of optional versus required fields under structured output enforcement can vary by SDK and model version. Confirm current behavior against the official OpenAI documentation, particularly around how the model is expected to represent an intentionally absent value.

A More Realistic Example: Structured Extraction with Nested Data

Real-world documents often contain repeated structures — a receipt has multiple line items, not just one total. Pydantic models can nest to represent this:

from typing import List, Optional
from pydantic import BaseModel


class LineItem(BaseModel):
    description: str
    price: float


class DetailedReceipt(BaseModel):
    merchant_name: str
    line_items: List[LineItem]
    total_amount: Optional[float] = None
    purchase_date: Optional[str] = None


response = client.responses.parse(
    model="gpt-5.6-terra",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Extract every line item with its price, plus the merchant name, total, and date.",
                },
                {
                    "type": "input_image",
                    "image_url": f"data:image/jpeg;base64,{image_b64}",
                    "detail": "high",
                },
            ],
        }
    ],
    text_format=DetailedReceipt,
)

receipt = response.output_parsed
for item in receipt.line_items:
    print(f"{item.description}: {item.price}")

LineItem models a single row of the receipt, and List[LineItem] in DetailedReceipt tells the schema that any number of these rows can appear — zero, one, or many. This nested structure mirrors the real shape of the data far more closely than a flat schema could, and it means your code can iterate over receipt.line_items directly with a simple for loop, exactly as it would over any other list of typed objects.

Testing Structured-Output Logic Without Calling the API

Because the object returned by output_parsed behaves like any normal Pydantic instance, code that processes it can be tested with a fake instance instead of a real API call:

def calculate_items_total(receipt: DetailedReceipt) -> float:
    return sum(item.price for item in receipt.line_items)


def test_calculate_items_total():
    fake_receipt = DetailedReceipt(
        merchant_name="Test Cafe",
        line_items=[
            LineItem(description="Latte", price=4.50),
            LineItem(description="Croissant", price=3.25),
        ],
        total_amount=7.75,
        purchase_date="2026-01-15",
    )
    result = calculate_items_total(fake_receipt)
    assert result == 7.75, f"Expected 7.75, got {result}"
    print("PASS: calculate_items_total sums line item prices correctly")


if __name__ == "__main__":
    test_calculate_items_total()

calculate_items_total takes a DetailedReceipt and has no idea whether it came from a real API call or was constructed by hand in a test — it just reads .line_items and sums the prices. The test builds a fake_receipt directly with known values and checks that the function produces the expected sum, entirely offline. This is the dependency-injection principle in action: the business logic (calculate_items_total) is decoupled from the data source (the API call that produced the original receipt), so it can be verified independently and cheaply.

Common Mistakes

Making every field required when some genuinely might not be visible in the image, which forces the model to fabricate a plausible-looking value rather than honestly reporting that a field is missing. Use Optional fields with a sensible default wherever a value might legitimately be absent from the source image.

Parsing response.output_text manually instead of using responses.parse with a schema, re-implementing fragile string parsing that structured output already solves more reliably. If you find yourself writing regular expressions to pull a number out of a sentence the model generated, that's a strong signal you should switch to a schema-based approach instead.

Designing a schema that doesn't match what's actually extractable from the image, such as requiring a field like tax_rate that receipts often don't display explicitly, forcing the model to either compute it (potentially incorrectly) or invent it. Design the schema around what a careful human could actually read off the image, not around what your downstream system would ideally like to have.

Best Practices

Keep schemas as flat as the data allows, reserving nested lists and objects for genuinely repeated or hierarchical data (like line items), since deeply nested schemas are harder for both you and the model to reason about correctly.

Make uncertain fields Optional and check for None downstream, rather than trusting that every field will always be populated with a correct value.

Reuse the same schema classes across both real requests and tests, as shown with DetailedReceipt and LineItem above, so your test fixtures stay in sync with your actual data contract instead of drifting into a separate, hand-maintained shape.