Vision Models Have Real, Practical Limits

It is tempting to treat a vision-capable model as an all-seeing oracle that can extract any information from any image you hand it. In practice, image understanding has firm boundaries, some technical (file size and format restrictions enforced by the API) and some perceptual (what is actually recoverable from a blurry or tiny image, which no model, human or artificial, can see past). Building a reliable application means designing for both kinds of limits explicitly, rather than discovering them in production when a user uploads an image that breaks your pipeline.

Technical Limitations: Size, Format, and Count

The API enforces concrete constraints on what an input_image part can contain:

• Supported formats are a specific, limited set of common image types (such as JPEG, PNG, WEBP, and non-animated GIF). Sending an unsupported format, like a raw TIFF or a proprietary camera format, will fail rather than being silently converted.
• Maximum file size for a single image is capped. An uncompressed high-resolution photo straight off a modern camera or phone can easily exceed this limit before you've done anything to it.
• Maximum number of images per request is capped as well, independent of your own good judgment about how many you should send (Lesson 7 covers the design reasoning; this is the hard ceiling on top of that).

Note: The exact supported formats, maximum file size, and maximum image count per request are specific values that can change between model versions and API updates. Confirm the current limits in the official OpenAI documentation before hardcoding assumptions about them into production validation logic.

Because these values can shift, the safest engineering pattern is to keep your own conservative limits in application code, configurable in one place, rather than scattering hardcoded numbers throughout your codebase:

import os

MAX_IMAGE_BYTES = 15 * 1024 * 1024  # conservative local cap; confirm current API limit separately
SUPPORTED_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".gif"}


def validate_image_file(path: str) -> None:
    extension = os.path.splitext(path)[1].lower()
    if extension not in SUPPORTED_EXTENSIONS:
        raise ValueError(f"Unsupported image format: {extension}")

    size_bytes = os.path.getsize(path)
    if size_bytes > MAX_IMAGE_BYTES:
        raise ValueError(
            f"Image is {size_bytes / (1024 * 1024):.1f} MB, "
            f"which exceeds the local limit of {MAX_IMAGE_BYTES / (1024 * 1024):.0f} MB"
        )

validate_image_file checks the extension against a known-good set and the file size against a locally defined ceiling, raising a clear ValueError with an actionable message in both failure cases. Running this check before encoding and sending the image means your application fails fast, locally, and cheaply — instead of paying for base64 encoding of a huge file only to have the API reject it after the fact.

Downscaling Large Images Before Sending

Rather than only rejecting oversized images, a more user-friendly application resizes them automatically when possible:

from PIL import Image
import io


def resize_if_needed(path: str, max_dimension: int = 2048) -> bytes:
    with Image.open(path) as img:
        width, height = img.size
        if max(width, height) <= max_dimension:
            with open(path, "rb") as f:
                return f.read()

        scale = max_dimension / max(width, height)
        new_size = (int(width * scale), int(height * scale))
        resized = img.resize(new_size, Image.LANCZOS)

        buffer = io.BytesIO()
        image_format = img.format if img.format else "JPEG"
        resized.save(buffer, format=image_format)
        return buffer.getvalue()

This function (using the third-party Pillow library, imported as PIL) opens the image, checks whether either dimension exceeds max_dimension, and only resizes if necessary — returning the original bytes untouched otherwise, which avoids a wasteful re-encode of images that are already small enough. When resizing is needed, it computes a single scale factor from the larger dimension so the image's aspect ratio is preserved, uses Image.LANCZOS as the resampling filter (a high-quality choice well suited to shrinking photographic images without introducing obvious artifacts), and writes the result into an in-memory io.BytesIO buffer rather than a temporary file, since the caller just needs the resulting bytes to base64-encode and send.

Why cap at a dimension like 2048 pixels rather than sending full camera resolution? Because, as discussed in Lesson 1, resolution beyond what the model actually uses for its internal processing adds token cost without adding usable information — the model cannot extract detail from pixels beyond what its vision encoder actually processes at a given detail level. Downscaling to a sensible ceiling before sending is one of the most effective, easy-to-implement cost optimizations available in a vision pipeline.

Perceptual Limitations: What No Model Can Recover

Some limitations aren't about API rules at all — they're about what information the image physically contains. No amount of prompting or model capability can recover:

• Text that is genuinely too small or blurry to resolve, even at maximum detail. If a human squinting at the same image on the same screen can't read a price, the model likely can't either.
• Information cropped out of the frame. If the total on a receipt is cut off at the edge of the photo, no model can report a number it was never shown.
• Content obscured by glare, shadow, or physical damage to the original document or object being photographed.
• Extremely low native resolution, such as a thumbnail-sized image scaled up — upscaling a small image before sending it does not add real detail; it only interpolates existing pixels into more pixels, which does not help the model recover information the source image never captured in the first place.

The correct engineering response to these cases is not to prompt harder — it's to build a path for the model to say "I cannot determine this from the image" and for your application to treat that response as valid, expected output that may require a fallback, such as asking the user to retake the photo or provide the missing value manually.

Designing for Graceful Degradation

A production-grade image pipeline should treat "the model couldn't read this" as a normal, expected outcome rather than an edge case that crashes the pipeline:

from typing import Optional
from pydantic import BaseModel


class ExtractionResult(BaseModel):
    value: Optional[str] = None
    confidently_extracted: bool


def interpret_extraction(result: ExtractionResult) -> str:
    if result.confidently_extracted and result.value is not None:
        return result.value
    return "MANUAL_REVIEW_REQUIRED"


def test_interpret_extraction_handles_low_confidence():
    uncertain_result = ExtractionResult(value="42.50", confidently_extracted=False)
    outcome = interpret_extraction(uncertain_result)
    assert outcome == "MANUAL_REVIEW_REQUIRED", f"Expected manual review flag, got {outcome}"
    print("PASS: low-confidence extraction is routed to manual review")


def test_interpret_extraction_accepts_confident_value():
    confident_result = ExtractionResult(value="42.50", confidently_extracted=True)
    outcome = interpret_extraction(confident_result)
    assert outcome == "42.50", f"Expected the extracted value, got {outcome}"
    print("PASS: confident extraction is passed through")


if __name__ == "__main__":
    test_interpret_extraction_handles_low_confidence()
    test_interpret_extraction_accepts_confident_value()

ExtractionResult includes a confidently_extracted boolean alongside the extracted value, so the model (prompted appropriately, as covered further in Lesson 9) can explicitly flag uncertain reads rather than mixing them in indistinguishably with confident ones. interpret_extraction then routes anything not both present and confident to a "MANUAL_REVIEW_REQUIRED" sentinel instead of passing a possibly-wrong guess further down the pipeline. The two tests validate this routing logic entirely offline, using hand-built ExtractionResult instances rather than a live model response, which is exactly the dependency-injection pattern used throughout this unit: the routing logic is proven correct independently of whether the actual extraction call behaves as expected.

Common Mistakes

Assuming higher detail can compensate for a fundamentally low-quality source image, and being surprised when results don't improve. detail controls how much of the image's existing information the model processes — it cannot invent detail the original image never captured.

Upscaling a small image before sending it, expecting better results, which only interpolates pixels without adding real information, and can occasionally make text look sharper to a human eye while not actually improving the model's ability to resolve it correctly.

Not validating file size and format before encoding, leading to wasted computation on the base64 encoding step for a file that was always going to be rejected, and a less clear error message than a local pre-check would have produced.

Best Practices

Validate format and size locally before sending any request, with clear, specific error messages, so failures are cheap and immediately actionable rather than surfacing as an opaque API error.

Downscale oversized images automatically rather than only rejecting them, improving the experience for users whose devices produce very large photos by default.

Build an explicit "could not confidently determine this" path into your data model and pipeline logic, and treat it as a normal, testable outcome rather than an exceptional failure — this is what separates a production-grade extraction pipeline from a demo that only works on clean example images.