Two Ways to Get an Image to the Model

Lesson 2 covered the URL form of input_image, where you pass a public link and the model's provider fetches the bytes itself. That approach only works when the image already lives somewhere reachable over the public internet. Most real applications don't have that luxury — the image is a file the user just uploaded from their phone, a screenshot captured locally, or a document sitting in a private database. For those cases, the SDK supports sending the image data directly, encoded as base64 text embedded in the request itself.

This lesson covers both paths in more depth than Unit 7 did, and gives you a clear rule for choosing between them.

Sending a Local File as Base64

To send a file you have on disk, read its bytes, encode them as base64, and build a data: URI string for the image_url field:

import base64
from openai import OpenAI

client = OpenAI()


def encode_image(path: str) -> str:
    with open(path, "rb") as image_file:
        encoded_bytes = base64.b64encode(image_file.read())
    return encoded_bytes.decode("utf-8")


image_base64 = encode_image("receipts/march_grocery.jpg")

response = client.responses.create(
    model="gpt-5.6-terra",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "What is the total amount on this receipt?"},
                {
                    "type": "input_image",
                    "image_url": f"data:image/jpeg;base64,{image_base64}",
                },
            ],
        }
    ],
)

print(response.output_text)

Walking through this example:

• encode_image opens the file in binary mode ("rb") because image bytes are not text, and text-mode reading would corrupt them by attempting character decoding.
• base64.b64encode converts the raw bytes into an ASCII-safe representation. Base64 exists because JSON — the format the SDK ultimately sends over HTTP — is a text-based format that cannot safely carry arbitrary binary bytes; base64 re-encodes those bytes as plain text characters that survive JSON encoding intact.
• The resulting string is prefixed with data:image/jpeg;base64,, forming what is called a "data URI." This prefix tells the receiving system two things: the MIME type of the content (image/jpeg) and the encoding used (base64). Without the correct MIME type prefix, the model's provider would not know how to decode the following bytes back into an image.
• The final image_url value looks like a URL syntactically, but no network request is made to fetch it — the image data is already embedded directly in the request payload.

Why Not Always Just Use a URL?

You could, in principle, upload every image to a public cloud bucket first and then send a URL. Many teams do this. But it adds infrastructure (a storage bucket, public access rules, a cleanup policy for temporary files) and a privacy consideration (temporarily public files, even under obscure paths, are not truly private). Sending base64 data directly avoids all of that: the image never needs to exist anywhere except in memory during the request. The trade-off is payload size — base64 encoding inflates the data by roughly 33%, and very large images increase request size and latency accordingly.

Choosing Between URL and Base64

A practical rule: if the image is already sitting at a stable, public URL, use that URL directly and skip encoding entirely — it is simpler and avoids inflating your request size. If the image originates from your user, your filesystem, or any private source, encode it as base64 and send it inline.

Handling User-Uploaded Files in a Web Application

A common real-world pattern is a web backend that receives an uploaded file (for example, through a form submission) and needs to forward it to the model without ever writing it to disk. You can encode the in-memory bytes directly:

import base64
from openai import OpenAI

client = OpenAI()


def analyze_uploaded_image(file_bytes: bytes, mime_type: str, question: str) -> str:
    encoded = base64.b64encode(file_bytes).decode("utf-8")
    data_uri = f"data:{mime_type};base64,{encoded}"

    response = client.responses.create(
        model="gpt-5.6-terra",
        input=[
            {
                "role": "user",
                "content": [
                    {"type": "input_text", "text": question},
                    {"type": "input_image", "image_url": data_uri},
                ],
            }
        ],
    )
    return response.output_text

This function takes the raw bytes your web framework hands you (for example, from an uploaded form field), along with the MIME type reported by the upload (such as "image/png" or "image/jpeg"), and the user's question. It never touches the filesystem, which is both faster and avoids leaving temporary files that need cleanup. Notice that the MIME type is a parameter here rather than hardcoded — different uploads can be PNG, JPEG, WEBP, or other formats, and the data URI must accurately declare which one it is, since an incorrect MIME type can cause the image to fail decoding on the receiving end even though the bytes themselves are fine.

Common Mistakes

Encoding the file in text mode instead of binary mode, i.e., opening with open(path, "r") instead of open(path, "rb"). Text mode applies character decoding to the bytes, which corrupts binary image data before it ever reaches base64.b64encode, typically causing an exception or a corrupted, unreadable image on the model's side.

Forgetting to decode the base64 bytes object back to a string, and passing the raw bytes object returned by base64.b64encode directly into an f-string or JSON payload. This produces a string with a b'...' wrapper visible in it. Always call .decode("utf-8") on the result before using it in the data URI.

Mismatching the declared MIME type and the actual file format — for example, hardcoding image/jpeg in the data URI prefix for a file that is actually a PNG. This is easy to do when the MIME type is copy-pasted from an earlier example rather than derived from the actual upload. Always use the MIME type that matches the real file format, ideally read from the upload metadata rather than assumed.

Best Practices

Prefer URLs for anything already hosted and stable, and reserve base64 encoding for genuinely private, local, or ephemeral images, since it keeps request payloads smaller and requests easier to debug.

Validate file size before encoding, so you can reject or downscale oversized images with a clear error message rather than sending an enormous base64 payload and receiving a cryptic failure from the API (image size and format limitations are covered in depth in Lesson 8).

Centralize your encoding logic in one helper function, like encode_image above, rather than duplicating base64-encoding code across every place in your application that sends an image — this makes it much easier to add validation, logging, or format checks in one place later.