Vision Models as an OCR Alternative

Optical character recognition (OCR) is the general term for pulling text out of an image. Traditional OCR engines work by detecting character shapes and matching them against known glyphs, which works well on clean, high-contrast, well-aligned text but tends to struggle with UI screenshots, handwriting, skewed photos, or dense mixed layouts like invoices and forms.

A vision-capable language model approaches the same problem differently: instead of only recognizing character shapes, it interprets the image the way a person would, understanding layout and context alongside the literal text. This makes it noticeably more capable at tasks like "read the total on this receipt" or "what does the error dialog say," because the model isn't just transcribing characters — it's also reasoning about which piece of text on the screen actually answers your question. This is genuinely new ground beyond what Unit 7 introduced: there, you learned how to attach an image at all; here, you use that capability specifically to pull structured, targeted information out of dense visual text.

A Basic Screenshot Extraction Example

import base64
from openai import OpenAI

client = OpenAI()


def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")


image_b64 = encode_image("screenshots/error_dialog.png")

response = client.responses.create(
    model="gpt-5.6-terra",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Transcribe all visible text in this screenshot exactly as it appears, preserving line breaks.",
                },
                {
                    "type": "input_image",
                    "image_url": f"data:image/png;base64,{image_b64}",
                    "detail": "high",
                },
            ],
        }
    ],
)

print(response.output_text)

Two details matter here that are specific to text-heavy images:

• detail is explicitly set to "high". Text extraction is exactly the kind of task that benefits from higher resolution processing, because small or dense text is the first thing lost when an image is downscaled. Leaving this at a low or default setting on a text-dense screenshot is one of the most common causes of missed or garbled words.
• The instruction asks for an exact transcription "preserving line breaks." Vision models, like text models, respond to how precisely you phrase the task. A vague instruction like "what does this say?" invites a summarized or paraphrased answer. If you need a faithful transcription rather than a summary, say so explicitly.

Extracting Specific Fields Instead of Full Text

Often you don't want the entire screen transcribed — you want particular fields, the way you'd extract data from a form. You can ask directly:

response = client.responses.create(
    model="gpt-5.6-terra",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": (
                        "This is a screenshot of an online order confirmation. "
                        "Extract only: order number, total price, and estimated delivery date. "
                        "If a field is not visible, respond with 'not found' for that field."
                    ),
                },
                {
                    "type": "input_image",
                    "image_url": f"data:image/png;base64,{image_b64}",
                    "detail": "high",
                },
            ],
        }
    ],
)

print(response.output_text)

The instruction to respond with "not found" for missing fields is deliberate and important. Without it, a model asked to extract a field that isn't actually present in the image may guess at a plausible-looking value rather than admit the field is absent — this is a form of hallucination, and it is far more dangerous in a data-extraction pipeline than a model simply saying it doesn't know. Explicitly telling the model what to do when information is missing closes off that failure mode and produces more trustworthy pipelines. (You'll take this further in Lesson 6, where structured output formats make missing fields impossible to represent ambiguously in the first place.)

Handling Dense or Tabular Content

Screenshots of tables, spreadsheets, or dashboards are harder than a single receipt because the spatial relationship between values matters — a number means nothing without knowing which row and column it belongs to. Give the model that structure explicitly in your instructions rather than assuming it will infer the ideal output format on its own:

response = client.responses.create(
    model="gpt-5.6-terra",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": (
                        "This screenshot shows a table of monthly expenses. "
                        "Reproduce it as a Markdown table with the same columns and rows. "
                        "If a cell is unreadable, write 'unclear' in that cell instead of guessing."
                    ),
                },
                {
                    "type": "input_image",
                    "image_url": f"data:image/png;base64,{image_b64}",
                    "detail": "high",
                },
            ],
        }
    ],
)

print(response.output_text)

Asking for a specific output format ("a Markdown table with the same columns and rows") gives the model a concrete target structure to preserve, rather than leaving it to decide how to linearize two-dimensional information into text — a decision that, left unconstrained, different runs might make differently. The "unclear" instruction serves the same anti-hallucination purpose as "not found" did in the previous example, adapted for a per-cell context.

When Screenshot Extraction Is the Wrong Tool

Vision-based extraction is not the right approach when:

• You control the data's original source. If a screenshot is being taken of a web page your own application rendered, it is almost always better to read the underlying data directly (from your database, your API response, the DOM) rather than round-tripping it through an image and asking a model to read it back out. This is faster, cheaper, and immune to misreading.
• You need guaranteed, deterministic accuracy on every character, such as extracting a legal document's exact wording for a compliance system. Vision models are highly capable but not infallible; a single misread digit in a legal or financial field can have serious consequences, and outputs should be validated or reviewed rather than trusted blindly in high-stakes settings.
• The image quality is too degraded for any method to read reliably — extreme blur, extreme low resolution, or heavy compression artifacts. No extraction technique, human or automated, can recover information that the image no longer visually contains (see Lesson 8 for more on quality limitations).

Common Mistakes

Leaving detail at a low or default setting for text-dense images, which silently degrades the model's ability to read small text, producing plausible-looking but subtly wrong transcriptions rather than an obvious failure. Always use detail="high" for screenshots and documents where exact text matters.

Asking an open-ended question instead of specifying the exact fields needed, which causes the model to decide on its own what's "important" to report, often omitting a field your downstream code actually depends on. Always name the exact fields or format you expect back.

Not instructing the model on what to do with missing or unclear information, which leaves the door open to confident-sounding guesses standing in for genuinely absent data. Always give an explicit instruction such as "write 'not found'" for fields that might not be present.

Best Practices

Crop or highlight the relevant region before sending the image when possible, especially for large screenshots where only a small part is relevant — this reduces ambiguity and often improves accuracy more than any prompt wording change would.

Always request an explicit fallback value for missing data ("not found," "unclear," null) rather than leaving the model free to decide how to represent absence.

Treat extracted text as untrusted input until validated, especially for numbers that feed into calculations or fields that trigger downstream actions — apply the same sanity checks you would to any OCR pipeline, such as format validation on dates, currency amounts, or IDs.