Fixing API Statelessness

The Direct Fix for Statelessness

Lesson 1 established the core fact you now need to build on: nothing about a conversation is remembered by the API unless you explicitly put it in the request. The most direct, most transparent, and most portable way to do that is what this lesson covers — manually building up a list of every message exchanged so far, and passing that entire list as input on every new call.

This technique doesn't rely on any special API feature. It works identically whether you're using the Responses API or the older Chat Completions API. It works regardless of what infrastructure you deploy on. And critically, it's the technique every other approach in this unit — previous_response_id in Lesson 3, the Conversations API in Lesson 4 — is ultimately built on top of, conceptually. Understanding this one pattern deeply makes everything after it easier to reason about, because you'll always know exactly what's happening under the hood, even when a more convenient API feature is doing some of the bookkeeping for you.

The Shape of a Message List

The Responses API accepts input as either a plain string (as you saw in Lesson 1) or as a list of message objects, each with a role and content:

messages = [
    {"role": "user", "content": "What's the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."},
    {"role": "user", "content": "What's the population there?"},
]

Three roles matter for a typical conversation:

• user — something the human said.
• assistant — something the model said in a previous turn. You include this so the model can see its own prior responses and stay consistent with what it already told the user.
• system (or developer, the Responses API's preferred name for the same concept) — instructions that set the model's behavior, tone, or constraints for the entire conversation, rather than being part of the back-and-forth exchange itself.

When you pass this full list as input, the model processes it exactly as if it were a single, continuous conversation — because from its perspective, in that one call, it is. The statelessness from Lesson 1 hasn't gone away; you're just explicitly re-supplying the full context on every call, which is precisely what's needed to simulate memory.

Setup

pip install openai python-dotenv

OPENAI_API_KEY=your_api_key_here

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

Building a Minimal Conversation Loop

Let's build the simplest possible version of a "chatbot that remembers" — a command-line loop that keeps a running list of messages and grows it after every exchange.

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

conversation_history = [
    {
        "role": "developer",
        "content": "You are a friendly, concise assistant. Keep answers under three sentences.",
    }
]

def send_message(user_text: str) -> str:
    # Add the user's new message to the running history
    conversation_history.append({"role": "user", "content": user_text})

    response = client.responses.create(
        model="gpt-5.6-luna",
        input=conversation_history,
    )

    assistant_text = response.output_text

    # Add the model's reply to the running history too, so future
    # calls can see what it already said
    conversation_history.append({"role": "assistant", "content": assistant_text})

    return assistant_text


print(send_message("My name is Priya. I'm building a customer support bot."))
print(send_message("What's my name?"))

Expected Output

Nice to meet you, Priya! A customer support bot is a great project —
let me know if you'd like help designing it.

Your name is Priya.

This is the fix. The second call succeeds — not because the model magically remembers, but because conversation_history now contains three items by the time the second send_message() call fires: the developer instruction, Priya's first message, the model's first reply, and then Priya's second message. All four of those items are sent together as input, giving the model everything it needs to answer correctly.

Code Walkthrough

conversation_history is a plain Python list, initialized with a single developer message that sets the assistant's behavior for the whole conversation. This is a natural place to put persistent instructions — tone, formatting rules, things the model should always keep in mind — since, unlike user and assistant messages, a developer message typically doesn't change turn to turn.

Inside send_message(), the very first thing that happens is appending the new user message to the list before calling the API. This ordering matters: the list passed to input needs to reflect the conversation exactly as it should appear to the model, ending with the newest thing said.

After getting the response back, the function appends the assistant's reply to the same list, using response.output_text — the convenience property that gives you the model's final text output directly, without you needing to dig through the more detailed response.output array yourself (useful for simple text-only conversations like this one; more complex responses involving tool calls need the fuller output array, which Lesson 2 of Unit 6 on function calling covers).

Because conversation_history is defined outside the function, it persists across calls to send_message() within the same running program — that's the actual mechanism providing the "memory." It's ordinary Python state, not anything special about the API.

Building This Into a Real Class

The loose function-and-global-list pattern above is fine for a quick demo, but for anything beyond a single script, you'll want conversation state properly encapsulated — especially once you're handling multiple simultaneous users, each with their own separate conversation.

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])


class Conversation:
    def __init__(self, system_prompt: str, model: str = "gpt-5.6-luna"):
        self.model = model
        self.history = [{"role": "developer", "content": system_prompt}]

    def send(self, user_text: str) -> str:
        self.history.append({"role": "user", "content": user_text})

        response = client.responses.create(
            model=self.model,
            input=self.history,
        )

        assistant_text = response.output_text
        self.history.append({"role": "assistant", "content": assistant_text})
        return assistant_text

    def reset(self) -> None:
        # Keep the system prompt, drop everything else
        self.history = self.history[:1]


# Usage
convo = Conversation(
    system_prompt="You are a helpful assistant for a bakery's online ordering system."
)
print(convo.send("Do you have gluten-free options?"))
print(convo.send("Great, can I order two of those for pickup tomorrow?"))

Expected Output

Yes! We offer a gluten-free chocolate chip cookie and a gluten-free
banana bread loaf. Would you like details on either?

I can help with that, but I'll need to know which item you'd like —
the gluten-free chocolate chip cookies or the banana bread loaf —
and what time tomorrow works best for pickup.

Notice the second reply correctly understands "those" refers back to the gluten-free options just discussed, and "tomorrow" as a pickup request — both only possible because self.history carried the first exchange into the second call. This Conversation class is a genuinely reusable building block: in a real application serving multiple users, you'd keep one Conversation instance per active user (or per chat thread), typically stored in a dictionary keyed by user ID or session ID, rather than a single shared list.

Where Real State Needs to Live: In-Memory vs. Persistent Storage

The examples above keep conversation_history as a Python variable, which only survives for as long as the process is running. That's fine for a script or a quick prototype, but it breaks the moment your application restarts, scales to multiple server processes, or needs a conversation to survive between a user's sessions.

For anything beyond a single-process demo, you need to persist the message list somewhere durable — the API itself, as Lesson 1 established, will not do this for you. Common approaches, roughly from simplest to most robust:

A dictionary keyed by session ID, for a single-process prototype where restarts and multiple servers aren't a concern yet. This is exactly what you'll build in Lesson 5's command-line chatbot project.

A database table, for anything that needs to survive a restart or run across multiple server instances — a conversations table holding a conversation ID, and a messages table holding each message's role, content, and a foreign key back to its conversation, ordered by a timestamp or sequence number. This is the standard pattern for a real production chatbot backend, and it's a natural fit for the FastAPI-based project you built (or will build) in the streaming and capstone units of this course.

A cache like Redis, when you want fast reads and writes for active conversations but don't necessarily need the long-term durability of a full database — often used alongside a database, where Redis holds the "hot" recent history for speed and the database holds the permanent record.

The choice depends entirely on what you're building. A single-user command-line tool is fine with an in-memory list. A multi-user web application needs at least a database table, because a Python list held in server memory disappears the instant that server process restarts, and won't be visible to a second server instance if your application scales horizontally.

Real-World Example: A Multi-User Support Widget

Consider a real deployment shape: a company embeds a chat widget on their website, and it needs to handle many simultaneous, independent conversations — one per visitor. Each visitor's browser is assigned a session ID (perhaps a cookie), and the backend maintains a mapping from session ID to that visitor's message history.

from collections import defaultdict

# In a real app this would be a database table, not an in-memory dict —
# shown here in-memory purely to illustrate the shape of the problem.
sessions: dict[str, list[dict]] = defaultdict(
    lambda: [{"role": "developer", "content": "You are a helpful shopping assistant for an online store."}]
)

def handle_incoming_message(session_id: str, user_text: str) -> str:
    history = sessions[session_id]
    history.append({"role": "user", "content": user_text})

    response = client.responses.create(
        model="gpt-5.6-luna",
        input=history,
    )

    history.append({"role": "assistant", "content": response.output_text})
    return response.output_text

Every incoming message is routed to handle_incoming_message() along with the visitor's session_id. Because each session ID maps to its own independent history list, visitor A's conversation about shipping policy never leaks into visitor B's conversation about a return — each one only ever sees its own accumulated history, because that's the only list ever passed as that session's input. This is the practical shape almost every production chatbot backend takes, whether it's built on a simple dictionary (fine for low traffic, single-process deployments) or a proper database (needed once you're running multiple server instances or need durability across restarts).

Trimming History Intelligently, Not Just Blindly

A natural question once you're actually maintaining growing history lists: do you always send the entire history, forever? In practice, no — for two reasons covered in depth in Lesson 6, but worth previewing here since it directly affects how you structure this code.

First, cost: every message in your history list gets billed as input tokens, on every single call, for as long as it stays in the list. A conversation that's grown to 200 messages means you're paying to re-process all 200 messages' worth of tokens on message 201, every time.

Second, the context window: the model has a hard maximum on how many tokens it can accept in a single request. A long enough conversation will eventually exceed that limit and cause an API error if you keep blindly appending forever.

A simple, immediately usable mitigation — full strategies are covered in Lesson 6 — is capping how many past turns you retain, while always keeping the system/developer message:

MAX_TURNS = 20  # keep the system message plus the last 20 user/assistant messages

def trim_history(history: list[dict]) -> list[dict]:
    system_message = history[0]
    recent_turns = history[1:][-MAX_TURNS:]
    return [system_message] + recent_turns

Calling trim_history(self.history) before constructing input keeps the payload bounded, at the cost of the model losing access to anything older than the last MAX_TURNS messages. Whether that tradeoff is acceptable depends entirely on your use case — Lesson 6 covers smarter strategies, like summarizing old turns instead of discarding them outright, for cases where losing old context entirely isn't acceptable.

Handling Errors Without Corrupting Your History

One detail that's easy to overlook until it bites you: what happens to your history list if the API call itself fails — a network timeout, a rate limit error, a temporary server error? If you append the user's message to history before the call, and the call then throws an exception, you're left with a history list that has a dangling user message with no corresponding assistant reply. The next successful call will look strange to the model, since it'll see two consecutive user messages with no response in between where one clearly should be.

A more robust version of the send() method guards against this:

def send(self, user_text: str) -> str:
    # Build the candidate history without mutating self.history yet
    candidate_history = self.history + [{"role": "user", "content": user_text}]

    try:
        response = client.responses.create(
            model=self.model,
            input=candidate_history,
        )
    except Exception:
        # The call failed — don't commit the user's message to history,
        # so a retry starts from a clean, consistent state.
        raise

    # Only commit both the user message and the assistant reply once
    # we know the call succeeded.
    self.history = candidate_history + [
        {"role": "assistant", "content": response.output_text}
    ]
    return response.output_text

This pattern — build a candidate version of your state, only commit it after confirming success — is a small change that prevents a real class of bugs from ever reaching production. It's especially worth doing once you've wired up retry logic (exponential backoff for rate limits, for example, which Unit 8's Moderation and reliability lesson covers), since a naive retry loop combined with an already-corrupted history list can compound the problem across multiple retries.

Including More Than Text: Images and Other Content Types

The examples so far have used plain strings for content, but a message's content can also be a list of content parts, which matters the moment your conversation needs to include something other than plain text — an image the user uploaded, for instance:

history.append({
    "role": "user",
    "content": [
        {"type": "input_text", "text": "What's wrong with this chart?"},
        {"type": "input_image", "image_url": "https://example.com/chart.png"},
    ],
})

The same principle from this entire lesson still applies unchanged: whatever you want the model to have access to in a later turn needs to be present in the input list you send. If a user uploads an image in turn one and asks a follow-up question about it in turn three, that image reference needs to still be present in the history list at turn three — the model doesn't retain any memory of images any more than it retains memory of text, and dropping a prior image from your trimming logic (from the section above) silently removes the model's ability to reference it in later turns, even if the conversation text around it is still present.

A Note on Cost: Why This Adds Up Faster Than It Looks

It's worth making the cost implication from earlier in this lesson concrete with real numbers, because it's easy to underestimate how quickly re-sending history compounds. Suppose each turn in a conversation averages 150 tokens (a reasonably chatty exchange). By the tenth turn, your history list contains roughly 1,500 tokens of accumulated context, and that entire 1,500 tokens gets re-sent, and re-billed as input tokens, on every single subsequent call — not just the newest 150-token message.

# Rough illustration, not exact pricing (check current rates before relying on this)
tokens_per_turn = 150
for turn_number in range(1, 21):
    history_tokens_this_call = tokens_per_turn * turn_number
    print(f"Turn {turn_number}: sending ~{history_tokens_this_call} tokens of history")

Turn 1: sending ~150 tokens of history
Turn 5: sending ~750 tokens of history
Turn 10: sending ~1500 tokens of history
Turn 20: sending ~3000 tokens of history

By turn 20, you're paying to reprocess roughly 20 times the tokens of a single isolated message, just to maintain conversational continuity. This is exactly the tradeoff Lesson 1 described: statelessness gives you full control, but that control comes with the responsibility of managing what gets re-sent, and unbounded history growth has a real, compounding cost. It's also worth knowing that OpenAI's API applies automatic prompt caching to repeated prefixes across calls, which meaningfully discounts the cost of re-sending identical leading content (like your system message and earlier turns that haven't changed) — but caching reduces this cost, it doesn't eliminate the fundamental scaling problem, which is why Lesson 6's trimming and compaction strategies still matter for genuinely long-running conversations.

Frequently Asked Questions

Do I need to include the developer/system message on every single call, or just the first one? Every single call. Remember Lesson 1's core lesson: nothing persists automatically. If you only include your system instructions on turn one and omit them from turn two onward, the model has no idea those instructions exist by turn two — it needs to see them, in full, in the input of every call where you want them to apply.

Can I edit or delete a message from history after the fact? Yes, and this is entirely your own list to manage however you like — since you own the data structure, you can remove a message, edit its content, or reorder entries (though reordering isn't usually meaningful, since conversations are inherently sequential) before the next call. A common real use case: redacting sensitive information from a stored history before it's used in a context where that data shouldn't be re-exposed.

What's the difference between system and developer roles? They serve the same conceptual purpose — standing instructions for the model's behavior — and the Responses API treats developer as its preferred, current terminology for what used to be called system in the Chat Completions API. Using developer is the more current convention for new Responses API code, though both are generally recognized.

Should every user message and every assistant reply be stored, or can I summarize as I go? For most conversations, storing every message verbatim is simplest and most reliable. Summarizing older turns instead of storing them verbatim is a valid strategy for very long conversations, but it's a deliberate tradeoff (you lose exact wording in exchange for a shorter context) covered properly in Lesson 6 — don't reach for it prematurely on a feature where conversations are naturally short.

Common Mistakes

Forgetting to append the assistant's reply back into the history. It's an easy step to skip, since the code "works" on the very next call regardless — but only the user's messages accumulate, and the model never sees its own prior answers. This produces subtly broken behavior: the model might repeat itself, contradict something it said earlier, or fail to build on a plan it previously laid out, because from its perspective, it never said those things.

Mutating a shared history list across unrelated users. If your application accidentally uses one global list (as in the first minimal example) for what should be multiple independent users' conversations, you'll get context bleeding across sessions — a serious bug and, depending on the content, a real privacy problem. Always key conversation state by session or user ID once you have more than one conversation happening.

Sending the system/developer message as a user message instead. Behavioral instructions belong in a developer (or system) role message, not folded into the first user turn. Mixing them together makes it harder for the model to distinguish "instructions I should always follow" from "something the user is asking about right now," and it also means your instructions get needlessly re-parsed as conversational content rather than standing instructions.

Never trimming history and being surprised by a context-length error months into production. A conversation that grows unbounded — support threads that go on for hundreds of messages, for example — will eventually exceed the context window if you always send the complete history. This tends to work fine in early testing (where conversations are short) and only breaks once real usage produces genuinely long threads, which is precisely the kind of failure that's easy to miss until Lesson 6's guidance is already needed in production.

Troubleshooting

The model responds as if it's forgotten something from three messages ago, even though I'm passing history. Print the exact input list right before the API call and check it actually contains what you expect. A common cause is a bug in the trimming logic (covered above) silently dropping more than intended, or a session-ID mismatch that's routing the message to the wrong history list.

I'm getting an error about the maximum context length being exceeded. This means your accumulated history, plus your new message, plus the model's expected output, exceeds that model's total context window. This is exactly the problem Lesson 6 addresses in full — for now, the immediate fix is applying the trimming approach above, or switching to a model with a larger context window if that fits your cost constraints.

Responses feel like they're drifting away from my system instructions over a long conversation. This isn't a bug in your history management — it's a known behavior where a model's adherence to instructions given early in a very long context can weaken relative to more recent content. If this becomes a real problem, consider re-stating key constraints periodically, rather than relying solely on a system message sent once at the very start of a now-very-long list.

Two different users are somehow seeing each other's conversation. This is almost always the shared-mutable-list bug described above. Audit your code for any conversation history that isn't clearly, explicitly keyed to one specific session or user.

Best Practices

Always key conversation state explicitly by session or user identity, even in early prototypes — retrofitting this later, once you have real shared state bugs to untangle, is far more painful than building it in from the start. Keep system/developer instructions in their own dedicated message at the front of the list, separate from the back-and-forth exchange, so your standing instructions and the live conversation don't get tangled together. Log the exact input sent on every call during development — it's the single most useful debugging tool available to you, precisely because of the transparency statelessness gives you (Lesson 1). And start thinking about history growth and trimming strategy early, even if you don't need it on day one — a conversational feature that works great in a demo with five-message exchanges can silently become broken, or expensive, the first time a real user has a two-hundred-message conversation with it.

Testing Conversation Logic Without Burning API Calls

Because Conversation.send() mixes pure history-management logic with an actual network call to OpenAI, it's worth separating the two conceptually when you write tests, so you can verify your history-management code is correct without needing a live API key or incurring real cost on every test run.

from unittest.mock import MagicMock

def test_history_accumulates_correctly():
    convo = Conversation(system_prompt="You are a test assistant.")

    # Replace the real API client with a stand-in that returns a fixed reply,
    # so this test never makes a real network call.
    fake_response = MagicMock()
    fake_response.output_text = "This is a fake reply."
    client.responses.create = MagicMock(return_value=fake_response)

    convo.send("First message")
    convo.send("Second message")

    assert len(convo.history) == 5  # system + 2 user + 2 assistant
    assert convo.history[1] == {"role": "user", "content": "First message"}
    assert convo.history[2] == {"role": "assistant", "content": "This is a fake reply."}

This test checks the part of your code you actually wrote and can break — whether messages get appended in the right order, with the right roles, at the right point in the flow — without depending on the model's actual output, which is naturally non-deterministic and shouldn't be something a unit test asserts against directly. Reserve real API calls for a smaller number of integration tests that specifically verify end-to-end behavior against the live API, and rely on this kind of mocked test for the history-management logic itself, which is exactly the part most likely to have a genuine bug (like the "forgot to append the assistant reply" mistake covered above) that has nothing to do with the model's actual responses.

What's Next

Manually managing message lists gives you full, transparent control — you always know exactly what's being sent, and the technique works identically across API providers and SDK versions. But it does mean you own all the bookkeeping: growing the list, storing it durably, trimming it when it gets too long. Lesson 3 introduces previous_response_id, a Responses-API-specific feature that lets OpenAI's servers handle some of this reconstruction for you, trading a bit of that manual control for real convenience — and it's worth understanding both approaches, because production systems often end up using a mix of the two depending on the situation.