Moving Beyond Text-Only Input

Every input to client.responses.create() so far in this course has been text — a string, or a list of role-tagged text messages (Unit 4's manual state pattern). The Responses API also accepts images as part of its input, letting a model see and reason about visual content directly, in the same request as any accompanying text instructions. This lesson covers input_image — the content type that carries an image into a request — including how images are supplied, what the model can and can't reliably do with them, and the practical considerations around size, format, and cost that come with adding a visual modality to a request.

The Shape of a Multimodal Input

Rather than passing input as a plain string, a request containing an image passes a list of content items — a mix of input_text and input_image entries — inside a message-shaped input structure.

response = client.responses.create(
    model="gpt-5.6-luna",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "What is shown in this image?"},
                {"type": "input_image", "image_url": "https://example.com/photo.jpg"},
            ],
        }
    ],
)
print(response.output_text)

This structure — a role of "user" and a content list mixing input_text and input_image items — is the multimodal generalization of the plain string input used everywhere earlier in this course; a plain string input is really shorthand for a single input_text item under the hood. Once an image needs to accompany text in a single request, the more explicit list-of-content-items structure becomes necessary, since a bare string has no way to carry anything beyond text.

Supplying an Image by URL

The simplest way to include an image is a publicly accessible URL, exactly as shown above — the model's infrastructure fetches the image from that URL at request time, meaning your application never needs to handle the image bytes directly at all.

def describe_image_from_url(image_url: str, question: str) -> str:
    response = client.responses.create(
        model="gpt-5.6-luna",
        input=[
            {
                "role": "user",
                "content": [
                    {"type": "input_text", "text": question},
                    {"type": "input_image", "image_url": image_url},
                ],
            }
        ],
    )
    return response.output_text

print(describe_image_from_url(
    "https://example.com/receipt.jpg",
    "What is the total amount on this receipt?",
))

This is the most convenient option when an image is already hosted somewhere accessible — a product photo already on a CDN, an image attached to a support ticket already stored in cloud storage — since no local file handling or encoding is needed on your end at all. It does depend on the URL being reachable by the model's infrastructure at request time, which matters for images behind authentication, on an internal network, or on a host with restrictive access controls (all covered next).

Supplying an Image as Base64-Encoded Data

When an image isn't available at a public URL — a file the user just uploaded to your application, a screenshot generated locally, an image behind authentication your application can reach but the model's infrastructure cannot — it can be embedded directly in the request as base64-encoded data, using a data: URL.

import base64

def describe_local_image(image_path: str, question: str) -> str:
    with open(image_path, "rb") as f:
        image_bytes = f.read()
    base64_image = base64.b64encode(image_bytes).decode("utf-8")

    # Determine the MIME type from the file extension — a simple approach;
    # a more robust version would inspect the actual file header
    ext = image_path.rsplit(".", 1)[-1].lower()
    mime_type = {"jpg": "image/jpeg", "jpeg": "image/jpeg", "png": "image/png", "webp": "image/webp"}.get(ext, "image/jpeg")

    response = client.responses.create(
        model="gpt-5.6-luna",
        input=[
            {
                "role": "user",
                "content": [
                    {"type": "input_text", "text": question},
                    {"type": "input_image", "image_url": f"data:{mime_type};base64,{base64_image}"},
                ],
            }
        ],
    )
    return response.output_text

print(describe_local_image("local_screenshot.png", "What error message is shown in this screenshot?"))

The data:{mime_type};base64,{base64_image} format is a standard data URL — the same mechanism browsers use to embed images inline in HTML or CSS — and it lets the entire image travel as part of the request payload itself, with no separate network fetch needed on the model's side. This is the right approach for any image your application has direct access to the bytes of but that isn't (and doesn't need to become) publicly hosted anywhere.

Why Image Size and Resolution Matter

Unlike text, where token cost scales roughly with word count, an image's cost and the level of visual detail the model can extract from it are governed by its resolution and how it's processed internally, which is worth understanding at a conceptual level even without needing to track the exact underlying mechanics precisely.

Note: The exact formula relating image resolution, tiling, and token cost is a platform-specific implementation detail that has changed across models and API versions historically, and is likely to continue evolving. Rather than memorizing a specific formula, treat this as a general principle — larger, higher-resolution images cost more and are typically processed with more visual detail available to the model — and confirm exact current pricing and resolution-handling behavior against your SDK version's documentation before building precise cost estimates into a production application.

def estimate_relative_image_cost(width: int, height: int) -> str:
    """Illustrative only — a rough intuition-builder, not a precise cost calculator.
    Larger images generally cost more tokens; extremely large images may be
    downscaled by the platform before processing regardless of what you send."""
    megapixels = (width * height) / 1_000_000
    if megapixels < 0.5:
        return "low — small image, likely processed at full detail cheaply"
    elif megapixels < 4:
        return "moderate — a typical photo or screenshot"
    else:
        return "higher — a large or high-resolution image, likely downscaled internally regardless"

The practical implication: sending an unnecessarily large image (a 12-megapixel photo when a 1-megapixel version would show the same relevant detail) typically wastes cost without improving the model's ability to answer questions about it, since the platform may downscale internally regardless — resizing an image to a reasonable resolution before sending it, covered next, is frequently the more cost-effective choice.

Resizing Images Before Sending Them

A common, practical preprocessing step is resizing an image to a reasonable maximum dimension before encoding and sending it, using a widely available image library.

from PIL import Image
import io
import base64

def prepare_image_for_request(image_path: str, max_dimension: int = 1024) -> str:
    """Resize an image so its largest dimension doesn't exceed max_dimension,
    then return it as a base64 data URL — reducing cost without meaningfully
    reducing the model's ability to read text or identify visual content."""
    img = Image.open(image_path)
    img = img.convert("RGB")  # normalize to avoid mode-related encoding issues

    if max(img.size) > max_dimension:
        ratio = max_dimension / max(img.size)
        new_size = (int(img.size[0] * ratio), int(img.size[1] * ratio))
        img = img.resize(new_size, Image.LANCZOS)

    buffer = io.BytesIO()
    img.save(buffer, format="JPEG", quality=85)
    encoded = base64.b64encode(buffer.getvalue()).decode("utf-8")
    return f"data:image/jpeg;base64,{encoded}"

data_url = prepare_image_for_request("large_photo.jpg")
response = client.responses.create(
    model="gpt-5.6-luna",
    input=[{"role": "user", "content": [
        {"type": "input_text", "text": "Describe this image."},
        {"type": "input_image", "image_url": data_url},
    ]}],
)

This preprocessing step is worth building into any application that accepts user-uploaded images at unpredictable sizes — a phone camera photo can easily be 12 megapixels or more, far beyond what's useful for most visual-understanding tasks, and resizing before sending both reduces cost and, in practice, often reduces request latency, since a smaller payload transmits and processes faster.

Multiple Images in One Request

More than one image can be included in a single request's content list, letting the model compare or reason across several images together — useful for tasks like comparing two product photos, reviewing a sequence of screenshots, or checking whether several images depict the same scene.

response = client.responses.create(
    model="gpt-5.6-luna",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Do these two product photos show the same item? Explain any differences."},
                {"type": "input_image", "image_url": "https://example.com/product_a.jpg"},
                {"type": "input_image", "image_url": "https://example.com/product_b.jpg"},
            ],
        }
    ],
)
print(response.output_text)

Each additional image contributes to the overall request's cost independently, following the same size-dependent logic as a single image — worth keeping in mind for a feature that might naturally want to include many images in one request (a full photo gallery, an entire multi-page scanned document), where cost can add up quickly across several full-resolution images and the resizing preprocessing from earlier in this lesson becomes proportionally more valuable.

What Vision Models Are Good and Not Good At

It's worth setting realistic expectations about the kinds of visual tasks a model handles reliably versus tasks it handles less reliably, since treating every visual task as equally trustworthy leads to disappointment on the harder end of this spectrum.

Generally reliable: describing the general content and composition of an image, reading clearly printed text (a sign, a document, a receipt with legible print), identifying well-known objects and scenes, comparing obviously different images, answering questions about color, general layout, or the presence/absence of a described element.

Less reliable, worth extra verification for anything consequential: precise counting of many small or overlapping objects, reading handwriting or heavily stylized or low-contrast text, precise spatial measurements or exact pixel-level positions, fine-grained distinctions between very similar-looking items (two nearly identical product variants), and any task requiring the model to notice something extremely subtle that a casual human glance might also miss.

def request_with_appropriate_caution(image_url: str, task_type: str) -> str:
    """A simple illustration of adjusting the request based on task difficulty —
    for less-reliable task types, explicitly ask the model to flag uncertainty."""
    if task_type == "precise_counting":
        instructions = "Count carefully. If items overlap or are hard to distinguish, say so explicitly rather than guessing a precise number."
    else:
        instructions = "Describe what you see clearly and concisely."

    response = client.responses.create(
        model="gpt-5.6-luna",
        instructions=instructions,
        input=[{"role": "user", "content": [
            {"type": "input_text", "text": f"Task: {task_type}"},
            {"type": "input_image", "image_url": image_url},
        ]}],
    )
    return response.output_text

For any application where a visual task falls into the "less reliable" category and the answer genuinely matters — a counting task feeding into inventory numbers, a measurement feeding into an automated decision — treating the model's visual output the same way Unit 6 treated free-text extraction (as something requiring validation, confidence checks, or human review for consequential decisions) is the right level of caution, rather than trusting a vision response with the same confidence as a well-defined, schema-constrained text extraction task.

Combining Images With Structured Outputs

Unit 6's structured-output mechanism works directly alongside image input — a schema constrains the shape of the answer regardless of whether the underlying content being analyzed is text, an image, or both together.

from pydantic import BaseModel

class ReceiptExtraction(BaseModel):
    merchant_name: str
    total_amount: float
    date: str

response = client.responses.parse(
    model="gpt-5.6-luna",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Extract the merchant name, total amount, and date from this receipt."},
                {"type": "input_image", "image_url": "https://example.com/receipt.jpg"},
            ],
        }
    ],
    text_format=ReceiptExtraction,
)
receipt = response.output_parsed
print(f"{receipt.merchant_name}: ${receipt.total_amount} on {receipt.date}")

This combination — visual input constrained to a structured schema — is directly useful for a large class of real applications: receipt and invoice processing, form digitization, ID or document verification workflows, all of which need both the model's ability to read an image and the reliability guarantee structured outputs provide over the resulting extracted data, exactly the same guarantee Unit 6 established for text-only extraction.

Handling Images That Fail to Load

A URL-based image can fail for reasons entirely outside the model's control — a broken link, an image behind authentication the model's infrastructure can't pass, a transient network issue fetching the URL — and this failure mode is worth handling explicitly rather than assuming every image URL will always resolve successfully.

def describe_image_safely(image_url: str, question: str) -> str | None:
    try:
        response = client.responses.create(
            model="gpt-5.6-luna",
            input=[{"role": "user", "content": [
                {"type": "input_text", "text": question},
                {"type": "input_image", "image_url": image_url},
            ]}],
        )
        return response.output_text
    except Exception as e:
        print(f"Failed to process image (URL may be unreachable or invalid): {e}")
        return None

For an application processing images from user-supplied or third-party URLs at any real volume, this kind of explicit error handling — rather than assuming success — is worth building in from the start, since a URL that was valid when a user submitted it can become unreachable by the time a request actually processes it (a temporary hosting outage, a deleted file, an expired signed URL), and the failure needs a clear, distinguishable signal rather than surfacing as a generic, unexplained request failure.

URL vs. Base64: A Direct Comparison

Both mechanisms for supplying an image ultimately deliver the same information to the model, but the practical trade-offs between them are worth laying out side by side rather than treating the choice as arbitrary.

Neither option is universally better — the right choice follows directly from where the image already lives. An application that already stores user uploads in cloud storage with public (or model-infrastructure-accessible) URLs should generally prefer the URL approach for its smaller request payloads; an application generating or receiving images that have no independent hosted existence (a screenshot captured in-process, a file freshly uploaded and not yet persisted anywhere) should use the base64 approach rather than adding an unnecessary hosting step purely to obtain a URL.

Combining Image Input With Conversation Memory

Images fit naturally into the conversation-memory mechanisms Unit 4 covered — a manually managed history list, in particular, can include past turns that contained images exactly as it includes past text turns, letting a multi-turn conversation reference an image shown earlier without needing to resend it on every subsequent turn.

history = [
    {
        "role": "user",
        "content": [
            {"type": "input_text", "text": "What's in this image?"},
            {"type": "input_image", "image_url": "https://example.com/chart.png"},
        ],
    },
]

response = client.responses.create(model="gpt-5.6-luna", input=history)
history.append({"role": "assistant", "content": response.output_text})

# A follow-up turn can refer back to the image without resending it,
# since the image already lives in the accumulated history
history.append({"role": "user", "content": "What's the highest value shown in that chart?"})
response = client.responses.create(model="gpt-5.6-luna", input=history)
print(response.output_text)

Whether an image genuinely stays available for reference across a previous_response_id chain (Unit 4, Lesson 3) rather than only in manually managed history is a detail worth confirming against your SDK version's current behavior — the safest, most broadly compatible approach for a conversation that needs to keep referring back to an earlier image is to include that image explicitly in the manually managed history list, exactly as shown above, rather than assuming any chaining mechanism automatically retains visual context from several turns back.

Testing Code That Handles Image Input

Following this course's consistent dependency-injection testing pattern, code that builds a multimodal input structure can be tested by asserting on the structure itself, without needing a live API call or a real image file for every test.

def build_image_question_input(image_url: str, question: str) -> list:
    """The logic under test — constructs the multimodal input structure."""
    return [
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": question},
                {"type": "input_image", "image_url": image_url},
            ],
        }
    ]

def test_build_image_question_input_structure():
    result = build_image_question_input("https://example.com/img.jpg", "What is this?")
    assert result[0]["role"] == "user"
    content = result[0]["content"]
    assert content[0] == {"type": "input_text", "text": "What is this?"}
    assert content[1] == {"type": "input_image", "image_url": "https://example.com/img.jpg"}
    print("PASS: build_image_question_input produces the correct multimodal structure")

test_build_image_question_input_structure()

Testing the structure-building logic this way — separately from the actual API call — is valuable specifically because a malformed content list (a missing type key, a URL placed under the wrong field name) is a common, easy-to-make mistake when hand-constructing these nested dictionaries, and a fast, free unit test catches exactly this class of error immediately, rather than only surfacing it as a confusing request-level failure after a live API call.

Common Mistakes

Sending unnecessarily large, unresized images, incurring higher cost and often higher latency without any corresponding improvement in the model's ability to answer questions about the image's content.

Treating every visual task as equally reliable, applying the same confidence to a precise counting or measurement task as to a straightforward "what's in this image" description, when the two categories have meaningfully different reliability profiles worth accounting for in any consequential application.

Assuming a URL-based image will always be reachable at request time, without handling the failure case explicitly, especially for images sourced from third parties or user-supplied links rather than infrastructure your own application controls.

Forgetting that multiple images in one request each contribute to cost independently, and not applying the same size-consciousness to a multi-image request that a single-image request would reasonably get.

Best Practices

Resize images to a reasonable maximum dimension before sending them, unless the specific task genuinely requires very fine visual detail, to reduce both cost and latency without meaningfully harming the model's ability to complete most visual tasks.

Use a data: URL for images your application already has direct access to, and a plain URL only for images that are already reliably, publicly hosted — don't add an unnecessary hosting step for an image your application could embed directly.

Match your confidence in a visual result to the task's actual reliability category, adding explicit uncertainty-flagging instructions or human review for tasks — precise counting, fine visual discrimination, measurement — known to be less reliable than straightforward description or well-printed text reading.

Combine visual input with structured outputs (Unit 6) for any extraction task, rather than parsing a free-text visual description, to get the same shape guarantees Unit 6 established for text-only extraction applied to image-derived data.

Handle image-loading failures explicitly for any application processing user-supplied or third-party image URLs, rather than assuming every URL will resolve successfully at request time.