Why Prompt Design Matters More, Not Less, With Images

Unit 3 covered prompting fundamentals for text: being specific, giving the model context, defining the output format, and avoiding ambiguity. Every one of those principles still applies when an image is involved, but images introduce an extra layer of ambiguity that text alone doesn't have — the model has to decide what in the image is relevant to your question, and a vague prompt gives it far more room to guess wrong than a vague text-only prompt would. "What's in this image?" invites a different kind of unhelpful answer than "summarize this" does, because there is often far more visually present in a photo than any single sentence could exhaustively describe, and the model has to choose what to emphasize.

This lesson is about closing that gap: writing multimodal prompts that produce the same useful answer reliably, across many similar images, rather than a good answer sometimes and an unhelpful one other times.

Principle 1: State the Task Before or Immediately After the Image, Never Only Implicitly

Compare these two prompts:

# Vague: relies on the model guessing what matters
content_vague = [
    {"type": "input_text", "text": "Here's a photo."},
    {"type": "input_image", "image_url": image_url},
]

# Specific: states exactly what's needed and in what form
content_specific = [
    {
        "type": "input_text",
        "text": (
            "This is a photo of a restaurant menu board. "
            "List every item name and its price, one per line, "
            "in the format 'Item Name - $Price'. "
            "If a price is not legible, write 'price unclear' instead."
        ),
    },
    {"type": "input_image", "image_url": image_url, "detail": "high"},
]

The vague version gives the model no task at all beyond implicitly inferring one from "here's a photo" — the model might describe the scene, guess at your intent, or produce a generic caption, and different runs of the exact same image could reasonably produce different kinds of answers. The specific version removes that ambiguity on three axes at once: what to extract (item names and prices), the exact output format (one per line, a specific template string), and what to do when information is missing (the "price unclear" fallback from Lesson 4). None of this is unique to vision prompting — it's the same specificity Unit 3 taught for text — but the effect of skipping it is more pronounced with images, because there's simply more raw visual information for an underspecified prompt to get lost in.

Principle 2: Tell the Model What It's Looking At

Vision models perform noticeably better when given brief context about the nature of the image, rather than being left to infer it purely from pixels. Stating "this is a screenshot of a mobile banking app" primes the model's interpretation before it starts reasoning about the content, the same way telling a person what they're about to look at helps them interpret it faster and more accurately than showing it cold.

content = [
    {
        "type": "input_text",
        "text": (
            "This is a screenshot of a mobile banking app's transaction history screen. "
            "Extract the five most recent transactions, each with its merchant name and amount."
        ),
    },
    {"type": "input_image", "image_url": image_url, "detail": "high"},
]

This is not redundant even when the image type seems visually obvious to you as the developer — the model has no other context about where this image came from, what application produced it, or what conventions that application follows (for instance, whether negative amounts represent debits or credits), unless you state it.

Principle 3: Constrain the Output Format Explicitly

An unconstrained natural-language answer is the least reliable format to build automation on top of, because its exact wording can vary between otherwise-identical requests. Whenever your downstream code needs to consume the answer programmatically, either use structured output (Lesson 6) or, at minimum, specify a strict textual template:

content = [
    {
        "type": "input_text",
        "text": (
            "Look at this photo of a parking sign. "
            "Answer with exactly one line in the format: "
            "'ALLOWED' or 'NOT ALLOWED', followed by a colon and a brief reason. "
            "Do not add any other text."
        ),
    },
    {"type": "input_image", "image_url": image_url},
]

"Do not add any other text" is doing real work here. Without it, a model might reasonably add a courteous preamble ("Based on the sign, it looks like...") before the actual answer, which is harmless for a human reader but breaks naive string parsing on the code side. When a rigid format truly matters, prefer structured output over a textual template like this one — a schema is enforced, while a textual instruction like this is a strong steering hint that the model reliably follows in the vast majority of cases but is not a hard guarantee the way a schema is.

Principle 4: Ask for Confidence or Uncertainty Explicitly

Building on Lesson 8's discussion of perceptual limits, you can prompt the model to self-report when it's uncertain, which is far more useful than a confident-sounding wrong answer:

content = [
    {
        "type": "input_text",
        "text": (
            "Read the expiration date printed on this food package. "
            "If the date is clearly legible, report it in YYYY-MM-DD format. "
            "If it is blurry, partially obscured, or ambiguous in any way, "
            "respond with 'UNCERTAIN' instead of guessing."
        ),
    },
    {"type": "input_image", "image_url": image_url, "detail": "high"},
]

This explicitly gives the model permission — and a clear instruction — to decline rather than guess. Models, like people put on the spot, will often produce a plausible-sounding answer when asked a direct question, even when the honest answer is "I'm not sure." Explicitly naming the uncertain case as an acceptable, expected response reduces this tendency meaningfully, and combined with a structured confidently_extracted field (Lesson 8), gives your application a real, machine-checkable signal for when a human should double-check the result.

Principle 5: One Task Per Request When the Task Is Complex

For genuinely complex extraction or analysis, a single sprawling prompt asking for many unrelated things at once tends to perform worse than either a well-organized single prompt with an explicit structured schema, or several focused smaller requests. Compare:

# Overloaded: too many disconnected asks bundled into one open-ended prompt
overloaded_prompt = (
    "Describe this image, list any text you see, identify all objects, "
    "guess the location, estimate the time of day, and comment on the mood."
)

# Focused: a single well-defined extraction task
focused_prompt = (
    "Identify every distinct object visible in this image. "
    "List each one on its own line, using short, specific noun phrases."
)

The overloaded prompt asks for six different, loosely related things in one breath, with no format guidance for any of them, making it likely that some parts get a thorough answer and others get a token, drive-by mention. The focused prompt does one thing well and specifies exactly how the answer should be organized. If you genuinely need all six of those pieces of information, define a structured schema with all six as fields (Lesson 6) rather than relying on a free-text answer to naturally organize itself.

A Prompt Design Checklist

Before sending a multimodal request in production code, check that your prompt:

• States the specific task, not just "look at this image."
• Gives brief context about what kind of image this is, if not obvious from the task itself.
• Specifies the exact output format needed, or uses a structured schema instead.
• Defines what the model should do when information is missing or unreadable.
• Asks for only one coherent task per request, or organizes multiple related asks into an explicit schema rather than an open-ended list.

Common Mistakes

Writing a prompt that would only make sense to someone who can already see the image, such as "is this okay?" with no stated criteria. The model has no shared context with you beyond exactly what you put in the request; ambiguous pronouns and unstated standards ("okay" by what measure?) produce unpredictable answers.

Assuming the model will always default to the most useful interpretation of a vague question, rather than accepting that vagueness is inherently unpredictable — different phrasings of the same underlying vague intent can and do produce meaningfully different answers.

Testing a prompt against only one example image and assuming it generalizes, when in practice minor variations between images (lighting, angle, layout) can expose weaknesses in an underspecified prompt that a single lucky test case didn't reveal. Test important prompts against a small, varied set of representative images before trusting them in production.

Best Practices

Write multimodal prompts with the same rigor as a structured output schema, even when you're not using one — specify the task, the format, and the fallback behavior every time, since these are exactly the ambiguities that cause inconsistent results.

Combine explicit prompt design with structured output whenever the answer feeds into code, using the prompt to guide the model's reasoning and the schema to guarantee the final shape of the answer.

Keep a small library of tested prompt templates for recurring tasks (receipt extraction, screenshot transcription, comparison tasks) rather than re-writing similar prompts ad hoc each time, so improvements you discover in one place propagate everywhere that template is reused.