What "Multimodal" Actually Means

A traditional language model request contains only text: a system instruction, a user question, maybe some prior conversation turns. A multimodal request can contain other kinds of data alongside that text — most commonly images, but depending on the model also audio or documents. The model is trained to interpret pixels the way it interprets tokens: as information it reasons about, not as an opaque attachment it merely passes along.

This distinction matters because it changes what your application can do. Instead of asking a user to describe a chart in words before you can help them interpret it, you can hand the model the chart image directly and ask your question. Instead of writing a separate OCR pipeline to pull text out of a scanned form, you can send the scan itself and ask the model to extract the fields you need. The model performs the "seeing" and the "reasoning" in a single pass.

In Unit 7, Lesson 1 ("Working with Images, Files, and Audio") you saw the basic mechanics of input_image — how to pass an image URL or a base64 string as part of a request. This unit goes further: it treats vision as a production capability, not a one-off trick. You will learn how to structure multimodal requests reliably, how to reason about image size and cost, how to extract structured data from images, how to work with multiple images at once, and how to design prompts that produce consistent results rather than occasional lucky guesses.

How the Responses API Represents Multimodal Content

With the OpenAI Python SDK, a request is built around the input parameter of client.responses.create(). When your input is plain text, you can pass a string directly. The moment you need to include an image, input becomes a list of messages, and each message's content becomes a list of typed content parts.

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-5.6-terra",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "What is shown in this image?"},
                {
                    "type": "input_image",
                    "image_url": "https://example.com/photos/receipt.png",
                },
            ],
        }
    ],
)

print(response.output_text)

Each content part has a type field that tells the model how to interpret that piece of data:

• "input_text" — a plain text fragment, equivalent to what you'd normally put in a simple string prompt.
• "input_image" — a reference to an image, either as a URL or as base64-encoded data (covered in Lesson 2 and Lesson 3).

Why does the SDK require this list-of-parts structure instead of just letting you pass an image alongside a string? Because a single user turn can legitimately contain several unrelated pieces of information — two images and a question about how they differ, for example — and the model needs an unambiguous, ordered way to receive them. A flat string cannot represent "here is text, then an image, then more text" without inventing a fragile markup convention. The typed list is the SDK's explicit, structured alternative to that markup.

Why the Model Needs role and type at All

Beginners sometimes ask why they can't just concatenate everything into one message. Two reasons:

1. role tells the model whose turn this is. A "user" role represents input from the person interacting with your application; a "system" role represents standing instructions you define at the developer level. Mixing these into a single undifferentiated blob would remove the model's ability to distinguish "the operator told me to behave this way" from "the end user is asking this."
2. type tells the model how to decode each content part. Text is tokenized directly. Images go through a separate vision encoder before being merged into the model's reasoning process. Without an explicit type, the SDK (and the model) would have no way to know which decoding path a given piece of content needs.

When to Use Multimodal Input — and When Not To

Vision input is powerful, but it is not free, and it is not always the right tool.

Use multimodal input when:

• The information genuinely lives in a visual medium: a screenshot, a scanned document, a photo of a whiteboard, a chart, a product photo.
• You want the model to reason jointly about visual and textual context, such as "does this photo match the product description below?"
• Building a text-extraction pipeline would require you to maintain a separate OCR or computer-vision service, and the model's built-in vision capability meets your accuracy requirements.

Avoid multimodal input when:

• The same information is already available to you as structured text or data. Sending a screenshot of a JSON payload instead of the JSON itself adds cost and risk of misreading for no benefit.
• You need pixel-perfect, deterministic extraction (for example, exact coordinates of a UI element for automated testing). Vision models describe and interpret; they do not guarantee exact geometric precision.
• Latency is critical and the visual content is decorative rather than informative — including it only slows down the request without improving the answer.

Images Are Not Free: Tokens and Cost

An image you send is converted internally into a number of tokens before the model processes it, and that number depends on the image's resolution and the detail level you request (detail is covered in depth in Lesson 8). This has two practical consequences:

• A single high-resolution image can consume more tokens than several paragraphs of text. If you are budgeting a context window or estimating cost per request, images must be accounted for explicitly, not treated as "free" attachments.
• Downscaling an image before sending it — for example, resizing a 4000×3000 pixel photo down to something closer to what the model actually needs to answer your question — can meaningfully reduce cost without meaningfully reducing answer quality, because the model does not need more resolution than is required to distinguish the relevant details.

Note: Exact image tokenization formulas and default detail behavior are model-specific and can change between model versions. Confirm the current values in the official OpenAI documentation before using them to plan production cost budgets.

A Minimal Mental Model for the Rest of This Unit

Keep this mental model as you move through the rest of the unit:

1. A multimodal request is a list of messages.
2. Each message has a role and a content list.
3. Each content part has a type that tells the model how to decode it — text, image URL, or image data.
4. Images cost tokens, and how many depends on resolution and detail level.
5. The model reasons over text and image content jointly, in the order the parts appear, not as separate isolated tasks.

Every lesson from here forward builds on this structure: sending images from different sources, extracting structured data from them, handling multiple images, and writing prompts that make the model's visual reasoning dependable rather than inconsistent.