Why Charts Come Back as Files, Not Text

A language model can describe a trend in words, but a chart communicates shape, magnitude, and outliers in a way prose cannot easily replicate. Code interpreter's sandbox has plotting libraries — matplotlib is the reliable default — pre-installed, so when the model decides a visual is the right way to answer a question, it writes code that renders a plot and saves it as an image file inside the sandbox filesystem. That file is then surfaced back to you as part of the response, addressable by its own file ID, separate from the text of the answer.

This is a meaningfully different retrieval path than the plain text answer you have used in every prior lesson, and getting it right is the difference between a feature that silently drops every chart it generates and one that reliably delivers them to your users.

Requesting a Chart

Ask for a chart the same way you would ask a person for one — describe what should be plotted and any formatting that matters:

from openai import OpenAI

client = OpenAI()

uploaded = client.files.create(file=open("monthly_revenue.csv", "rb"), purpose="assistants")

response = client.responses.create(
    model="gpt-5.6-terra",
    tools=[{
        "type": "code_interpreter",
        "container": {"type": "auto", "file_ids": [uploaded.id]},
    }],
    input=(
        "monthly_revenue.csv has columns month and revenue. Create a line "
        "chart of revenue over month, with the month on the x-axis, a "
        "clear title, and labeled axes. Save it as revenue_trend.png."
    ),
)

Two phrasing choices matter here. First, specifying the chart type (line chart) rather than leaving it to the model's judgment gives you a predictable, reviewable result — for a time series, a line chart is the right call, but for categorical comparisons you would ask for a bar chart explicitly rather than hoping the model picks the one you had in mind. Second, naming the output file (revenue_trend.png) gives you a known filename to look for when extracting it from the response, which the next section relies on.

Extracting the Generated Image

The image the model created lives inside the sandbox's filesystem and is exposed through the response as a generated file, referenced by a file ID that appears on the code_interpreter_call output item.

image_file_id = None
for item in response.output:
    if item.type == "code_interpreter_call":
        for result in item.outputs or []:
            if result.type == "image":
                image_file_id = result.file_id

if image_file_id:
    file_content = client.containers.files.content(
        container_id=item.container_id,
        file_id=image_file_id,
    )
    with open("revenue_trend.png", "wb") as f:
        f.write(file_content.read())
    print("Chart saved locally.")
else:
    print("No image was generated in this response.")

Walking through this: the loop first finds the code_interpreter_call item and inspects its outputs — the list of results the sandbox execution produced, which can include text, images, or other file references depending on what the code did. When an entry's type is image, its file_id is the handle needed to actually download the bytes. That download happens through a container-scoped files endpoint, because the generated file lives inside that specific sandbox container rather than in your account's general file storage (contrast this with the files you uploaded yourself in Lesson 3, which use the plain client.files resource). The downloaded content is a binary stream, written to disk in binary mode, the same way an uploaded file is read in binary mode.

Note: The exact structure of item.outputs, the discriminator value used for image results (shown here as "image"), and the specific method used to download a container-generated file (client.containers.files.content here) are all details that can change between SDK versions. Confirm the current output schema and download method against the official Responses API and SDK reference before depending on these exact names in production code.

Generating a Data Summary Alongside a Chart

Charts and numeric summaries are usually more useful together than either alone, and a single request can produce both:

response = client.responses.create(
    model="gpt-5.6-terra",
    tools=[{
        "type": "code_interpreter",
        "container": {"type": "auto", "file_ids": [uploaded.id]},
    }],
    input=(
        "Using monthly_revenue.csv: (1) create a bar chart of revenue by "
        "month, (2) report the month with the highest revenue and the "
        "month with the lowest, (3) report the average month-over-month "
        "growth rate as a percentage, rounded to one decimal place."
    ),
)

print(response.output_text)

Numbering the three deliverables in the prompt is a small technique with an outsized effect: it gives the model an explicit checklist, which reduces the chance that one part (commonly the numeric summary, when a chart is also requested) gets skipped or answered only partially. When you need every part of a multi-part request reliably, structuring the request as a numbered list is more effective than a single flowing sentence describing several things at once.

Choosing the Right Chart Type

Part of writing a good prompt is knowing, yourself, what chart type actually fits the question — the model will follow your instruction, but a wrong instruction produces a technically correct, uselessly wrong chart.

If you are not sure which chart fits, you can ask the model to decide and explain its choice rather than specifying one — "choose the chart type that best shows how spending is distributed across categories, and explain why you picked it" is a legitimate and often effective prompt when exploratory flexibility matters more than a predictable output format.

Handling Requests That Produce No Chart

Not every analysis request results in a generated file — a purely numeric question ("what's the average?") has no reason to produce an image, and code that only prints a value will not populate an image entry in outputs. Production code should treat a missing chart as an expected, handled case rather than an error:

def extract_chart(response) -> bytes | None:
    for item in response.output:
        if item.type != "code_interpreter_call":
            continue
        for result in item.outputs or []:
            if result.type == "image":
                content = client.containers.files.content(
                    container_id=item.container_id,
                    file_id=result.file_id,
                )
                return content.read()
    return None


def test_extract_chart_returns_none_when_no_image():
    class FakeItem:
        type = "code_interpreter_call"
        outputs = []
        container_id = "cnt_fake"

    class FakeResponse:
        output = [FakeItem()]

    assert extract_chart(FakeResponse()) is None
    print("PASS: extract_chart returns None when no image is present")


test_extract_chart_returns_none_when_no_image()

This test follows the dependency-injection style used throughout this course: instead of making a real API call, it constructs plain fake objects (FakeItem, FakeResponse) that mimic the exact shape extract_chart reads from, and asserts the function behaves correctly against that shape without needing network access or a real API key. This is the right way to test extraction logic — the logic under test is pure Python parsing a response structure, and it deserves a fast, deterministic test independent of whether the live API is reachable.

Common Mistakes

Assuming every response contains a chart because a chart was requested. If the model determines the request doesn't actually require a visual, or if generated code encounters an error, outputs may not contain an image entry at all. Always check before assuming, as shown above, rather than indexing into a list that might be empty.

Downloading the generated image using the wrong file resource. A generated file lives in the container that produced it, not in your account's general uploaded-files store — attempting to fetch it with the plain files endpoint used for uploads (Lesson 3) will not find it, because it is scoped to the container.

Requesting many charts in a single prompt without naming each one. "Show me a few charts about this data" produces an unpredictable number of loosely defined charts. If you need more than one, enumerate exactly what each one should show.

Best Practices

Always specify the chart type explicitly when you know what you want to see — treat "the model's judgment" as the right choice only for genuinely exploratory requests, not for dashboard features where consistent chart types matter for the user experience.

Save generated images with a deliberate naming and storage scheme in your own application (a database record linking a user, a request, and a file path), since container-generated files are not permanently retrievable from OpenAI's side once the container expires — extraction has to happen at request time.

Write unit tests for your extraction and parsing logic using fake response objects, not live API calls, so that logic can be verified quickly and deterministically as part of normal development, independent of network access or API cost.