Multimodal Chain-of-Thought Prompting

What Is Multimodal Chain-of-Thought? A modality is a type of input: text, images, audio, or video. A multimodal model can accept more than one modality, for example a question in text together with a photo, chart, or screenshot.

Multimodal chain-of-thought (multimodal CoT) applies step-by-step reasoning (Lesson 39) to problems that combine text and images. The key idea is to separate two stages:

Observe: describe or extract what the image actually shows. Reason: use those observations, together with the text, to reach an answer. Image + question ↓ Stage 1: What exactly does the image show? (observations) ↓ Stage 2: Given these observations, what is the answer? (reasoning) ↓ Answer Where the Idea Comes From The 2023 paper "Multimodal Chain-of-Thought Reasoning in Language Models" by Zhang et al. proposed a two-stage framework: first generate a rationale using both the text and the image, then infer the answer from that rationale. Their models were comparatively small (under one billion parameters) and fine-tuned for the task, and they reported strong results on the ScienceQA benchmark, which contains science questions with images.

Today, many general-purpose models accept images directly through their APIs. The same principle, observing first and reasoning second, can be applied through prompting, without any fine-tuning. That practical use is the focus of this lesson.

Why Separate Observation From Reasoning? Errors Come From Two Different Places A wrong answer to an image question can have two very different causes:

Error Type Example Perception error The model reads a bar as 45 when it shows 54, or misses a small label Reasoning error The model reads the values correctly but compares or calculates them wrongly When the model answers directly, you cannot tell which kind of error happened. When it writes its observations first, you can inspect them: if the observations are wrong, the problem is perception; if they are right but the answer is wrong, the problem is reasoning.

Observations Ground the Reasoning Without explicit observations, a model may answer from general expectations rather than from the image. For example, asked "Which month had the most enrollments?", it might answer with a typical peak month instead of reading the chart. Writing observations first ties the reasoning to what is actually in the image.

Observations Can Be Checked Structured observations, such as a list of values read from a chart, can be validated in code (Lesson 73), shown to a user for confirmation, or compared with other data sources.

Prompt Patterns for Multimodal CoT Pattern 1: Observe, Then Answer (Single Prompt) Look at the chart in the image.

Step 1 — Observations: list every value shown in the chart, with its label, exactly as displayed. If a value is not readable, write "unreadable". Step 2 — Reasoning: using only your observations, answer the question. Step 3 — Answer: give the final answer on one line.

Question: Which month had the largest increase in enrollments compared with the previous month? Why It Works "Exactly as displayed" discourages estimating or rounding. "Unreadable" gives the model a way out instead of guessing (Lesson 10, Tip 3). "Using only your observations" prevents the answer from relying on general knowledge. Pattern 2: Structured Observations For data in images, such as charts, tables, receipts, and forms, request the observations as JSON:

Extract the data from the bar chart in the image as JSON: {"title": string, "unit": string or null, "bars": [{"label": string, "value": number}]} Copy labels exactly. Use null for any value you cannot read clearly. Return only the JSON. The JSON can then be validated and processed in code, for example to compute increases exactly (Lesson 58).

Pattern 3: Region-by-Region Description For complex images such as screenshots or diagrams, ask the model to describe regions in a fixed order before answering:

This is a screenshot of a web application's error page. First, describe each part of the screen from top to bottom: header, main message, any error codes, buttons, and small text. Then, based only on that description, explain the most likely cause of the error and what the user should do. Describing parts in order makes it less likely that small but important details, such as an error code in small text, are missed.

Two-Stage Multimodal CoT in Python The following example separates the stages into two requests: the first extracts observations from a chart as JSON, code validates them and computes the answer exactly, and a second request explains the result.

Requirements: pip install openai and an OPENAI_API_KEY environment variable. The model you use must accept image input; check your provider's model documentation.

Preparing an Image import base64 import mimetypes from pathlib import Path

def image_to_data_url(path: str) -> str: """Encode a local image file as a data URL that can be sent to the API.""" mime_type, _ = mimetypes.guess_type(path) if mime_type not in {"image/png", "image/jpeg", "image/webp", "image/gif"}: raise ValueError(f"Unsupported image type: {mime_type}") encoded = base64.b64encode(Path(path).read_bytes()).decode("ascii") return f"data:{mime_type};base64,{encoded}" Code Explanation mimetypes.guess_type() determines the file type from its extension, such as image/png. The function rejects file types that are not common image formats, so a wrong file fails early with a clear message. base64.b64encode() converts the binary image into text, and the result is wrapped in a data URL, a standard way to embed file contents directly in a string: data:image/png;base64,.... Stage 1: Extract Observations import json

from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with an image-capable model available in your account

OBSERVE = """Extract the data from the bar chart in the image as JSON: {"title": string, "unit": string or null, "bars": [{"label": string, "value": number or null}]} Copy labels exactly as shown. Use null for any value you cannot read clearly. Return only the JSON."""

def observe_chart(image_path: str) -> dict: response = client.responses.create( model=MODEL, input=[{ "role": "user", "content": [ {"type": "input_text", "text": OBSERVE}, {"type": "input_image", "image_url": image_to_data_url(image_path)}, ], }], ) return json.loads(response.output_text) Code Explanation The request contains one user message with two content parts: an input_text part with the instructions and an input_image part with the image as a data URL. This is how the OpenAI Responses API accepts images together with text. Images can also be given as public https:// URLs instead of data URLs. The prompt asks only for observations, in a structured format. It does not ask the question yet, so the observations are not influenced by an expected answer. Validation and Exact Computation in Code def largest_increase(observations: dict) -> tuple[str, float]: bars = observations["bars"] if any(bar["value"] is None for bar in bars): raise ValueError("Some values were unreadable; ask for a clearer image or a manual check") changes = [ (bars[i]["label"], bars[i]["value"] - bars[i - 1]["value"]) for i in range(1, len(bars)) ] return max(changes, key=lambda change: change[1])

sample_observations = { "title": "Monthly enrollments", "unit": "students", "bars": [ {"label": "Jul", "value": 120}, {"label": "Aug", "value": 135}, {"label": "Sep", "value": 190}, {"label": "Oct", "value": 205}, ], } print(largest_increase(sample_observations)) Output:

('Sep', 55) Code Explanation If any value was unreadable (None), the function refuses to guess and asks for human attention. The month-to-month changes are computed exactly: Aug +15, Sep +55, Oct +15. September has the largest increase. The observations above are a sample, so the logic can be tested without an image or API call. In the full workflow, they come from observe_chart(). Stage 2: Explain the Result EXPLAIN = """Using only the data in and the computed result in , explain in two sentences which month had the largest increase in enrollments and by how much."""

def explain(observations: dict, result: tuple[str, float]) -> str: response = client.responses.create( model=MODEL, instructions=EXPLAIN, input=f"\n{json.dumps(observations)}\n\n\n{result}\n", ) return response.output_text The explanation step does not see the image at all. It works only from validated observations and an exact result, so it cannot reintroduce perception errors.

Verifying What the Model Saw Perception errors are the main risk in multimodal tasks. Practical ways to reduce them:

Technique How It Helps Ask for observations before answers Makes perception errors visible Request exact values and allow "unreadable" Discourages guessing Check observations against known constraints For example, percentages in a pie chart should total about 100 Compare with another source For example, compare values read from a screenshot with the database Use higher-quality images Small text and low resolution increase reading errors Show observations to the user for confirmation Lets a person catch misreadings before decisions are made Example Constraint Check For a receipt, the line items read from the image should add up to the printed total. If they do not, at least one value was misread, and the result should be flagged rather than used.

Common Multimodal Use Cases Use Case Observation Stage Reasoning Stage Chart questions Extract labels and values Compare, compute in code, explain Error screenshots Describe message, code, and context Diagnose cause and suggest fixes Handwritten or printed forms Extract fields Validate and fill a database record UI design review Describe layout and elements Evaluate against accessibility or design guidelines Science or exam diagrams Describe parts and labels Answer the question using the description Common Mistakes Mistake 1: Asking for the Answer Directly What it is: "What is the total on this receipt?" without asking for observations.

Why it happens: It is the shortest prompt.

What it causes: Perception and reasoning errors cannot be distinguished, and the model may answer from expectations.

How to fix it: Ask for observations first, preferably in a structured format.

Mistake 2: Trusting Values Read From Images What it is: Using numbers read from a chart or document without checking.

Why it happens: The model reports them confidently.

What it causes: Misread digits lead to wrong calculations and decisions.

How to fix it: Validate with constraints, compare with other sources, and allow "unreadable".

Mistake 3: Calculating in the Model What it is: Asking the model to read values and compute results in one response.

Why it happens: It seems efficient.

What it causes: Arithmetic errors on top of possible reading errors.

How to fix it: Extract values with the model and compute in code (Lesson 58).

Mistake 4: Sending Unsuitable Images What it is: Low-resolution, cropped, or very cluttered images.

Why it happens: Images are used as they arrive.

What it causes: More reading errors and more "unreadable" values.

How to fix it: Request clear images, crop to the relevant region, and check image quality before processing.

Mistake 5: Ignoring Privacy in Images What it is: Sending screenshots or documents that contain personal information not needed for the task.

Why it happens: Images contain more than the relevant part.

What it causes: Unnecessary exposure of personal data.

How to fix it: Crop or mask sensitive areas before sending images.

Best Practices Observe first, reason second. Separate perception from reasoning.

Request structured observations for data in images. JSON can be validated and computed.

Allow "unreadable". Never force the model to guess a value.

Compute exact results in code. Then let the model explain them.

Validate observations against constraints and other data. Catch misreadings early.

Keep the reasoning step away from the raw image when possible. Work from validated observations.

Protect privacy. Send only the image content the task requires.

Try It Yourself Exercise 1: Observe vs Answer Directly Create a bar chart with five values using a spreadsheet or plotting tool. Ask an image-capable model three questions about it, first directly and then with the observe-then-answer pattern. Compare the observations with the real values and check where any errors came from.

Exercise 2: Receipt Check Photograph or create an image of a simple bill with four items and a total. Extract the items and total as JSON, check in code whether the items sum to the total, and decide what your application should do when they do not match.

Exercise 3: Screenshot Diagnosis Take a screenshot of an error message from a development tool you use. Use Pattern 3 (region-by-region description) to describe the screen and then diagnose the error. Check whether the description captured every important detail, including error codes and file names.