Conversation Memory Challenges

The Problem Every Conversation Eventually Hits

Every technique in this unit — manual history (Lesson 2), response chaining (Lesson 3), the Conversations API (Lesson 4), and the chatbot project (Lesson 5) — shares one unavoidable limitation: the model can only process a finite amount of text in a single call. That limit is called the context window, and every conversation that runs long enough will eventually approach it, regardless of which state-management technique is holding that conversation's history.

This lesson covers three things you need in order to handle that limit gracefully instead of discovering it as a production error: understanding exactly what counts toward the context window, counting tokens before you hit the limit rather than after, and applying trimming or compaction strategies so a long conversation degrades gracefully instead of failing outright.

What the Context Window Actually Is

A model's context window is the maximum number of tokens — input, output, and, for reasoning models, internal reasoning tokens — that can be involved in a single request. This isn't three separate limits you manage independently; it's one shared budget. If a model has a 400,000-token context window and your conversation history uses 395,000 tokens, you have only 5,000 tokens left for the model's own output — even if that's nowhere near enough for a complete answer, the model has to work within whatever's left.

This is a hard limit, not a soft guideline. Exceed it, and the API returns an error rather than silently truncating your input for you. This matters because it means context-length problems aren't a subtle quality-degradation issue you might not notice — they're a hard failure that will break your application the moment a conversation crosses the line, if you haven't planned for it.

What Counts Toward the Limit

It's worth being precise about what actually consumes context-window budget, since it's easy to underestimate:

Every message in your conversation history — every user, assistant, and developer/system message you've included, whether via Lesson 2's manual list, Lesson 3's response chain, or Lesson 4's Conversations API. This is the part most people think of first, and it's usually the biggest contributor in a long-running conversation.

Tool/function definitions, if you're using function calling (Unit 6) — the full JSON schema of every tool you make available to the model on a given call counts as input tokens, every single call, regardless of whether the model actually decides to use any of them.

Any file or document content you've included directly in the conversation — pasted text, retrieved documents for RAG-style features, and similar.

The model's own output, including any internal reasoning tokens for reasoning-capable models like gpt-6-astra. Reasoning tokens in particular can be substantial and aren't always visible to you directly, which is worth remembering when budgeting headroom for a reasoning model's response.

Setup

pip install openai tiktoken python-dotenv

OPENAI_API_KEY=your_api_key_here

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

Counting Tokens Before You Send a Request

Rather than discovering you've exceeded the context window from an API error, you can estimate token counts locally using tiktoken, OpenAI's own tokenizer library, and use that estimate to decide whether trimming is needed before you make the call at all.

import tiktoken

def count_tokens(text: str, model: str = "gpt-5.6-luna") -> int:
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        # Fallback for a model tiktoken doesn't have a specific mapping for
        encoding = tiktoken.get_encoding("cl100k_base")
    return len(encoding.encode(text))


def count_history_tokens(history: list[dict]) -> int:
    total = 0
    for message in history:
        # A rough overhead per message for role/formatting metadata,
        # in addition to the content itself
        total += 4
        total += count_tokens(message["content"])
    return total


history = [
    {"role": "developer", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."},
]

print("Estimated tokens:", count_history_tokens(history))

Expected Output

Estimated tokens: 33

Code Walkthrough

tiktoken.encoding_for_model() returns the specific tokenizer associated with a given model name; encoding.encode(text) converts a string into the list of integer token IDs the model would actually process, and the length of that list is the token count. The fallback to cl100k_base handles the case where tiktoken doesn't have a specific mapping for a newer or custom model name — a reasonable general-purpose encoding to estimate with when an exact mapping isn't available.

count_history_tokens() sums the token count of every message's content, plus a small fixed overhead per message to roughly account for the role label and structural formatting each message carries — this won't be perfectly exact (the real count depends on internal formatting details of the API request itself), but it's more than close enough to use as a practical budget check before sending a request.

Checking Budget Before Every Call

With a token-counting function in hand, you can build a simple guard that checks remaining budget before making a call, rather than finding out you've exceeded it from an error:

CONTEXT_WINDOW = 400_000  # check current documentation for your specific model
RESERVED_FOR_OUTPUT = 4_000  # leave headroom for the model's response

def has_room_for(history: list[dict], model: str = "gpt-5.6-luna") -> bool:
    used = count_history_tokens(history)
    return used < (CONTEXT_WINDOW - RESERVED_FOR_OUTPUT)


if not has_room_for(history):
    print("Conversation is approaching the context limit — trimming needed.")

This pattern — estimate, then decide whether to act, before making the actual API call — is the core discipline this lesson is built around. It turns an unpredictable hard failure into a predictable, testable condition your code can check for and handle deliberately.

Strategy 1: Simple Truncation

The most straightforward mitigation, previewed in Lesson 2, is dropping the oldest messages once history grows past a threshold, while always preserving the system/developer message:

def truncate_to_recent(history: list[dict], max_messages: int = 40) -> list[dict]:
    system_message = history[0]
    recent = history[1:][-max_messages:]
    return [system_message] + recent

This is simple, predictable, and cheap to compute — but it's a blunt instrument. It discards old context entirely, with no attempt to preserve anything useful from what's dropped. For a conversation where old context genuinely stops mattering (a quick support interaction, for instance, where only the last few exchanges are relevant to the current question), this is perfectly adequate. For a conversation where something important was established early — a stated constraint, a key fact the user shared — blind truncation can silently lose it, producing exactly the kind of "why did it forget that" confusion Lesson 1 taught you to recognize, just from a different root cause.

Strategy 2: Summarization

A smarter approach: instead of discarding old messages outright, periodically summarize them into a compact form and replace the discarded originals with that summary, preserving the gist while shrinking the token count substantially.

def summarize_old_messages(old_messages: list[dict]) -> dict:
    transcript = "\n".join(
        f"{m['role']}: {m['content']}" for m in old_messages
    )

    summary_response = client.responses.create(
        model="gpt-5.6-luna",
        input=[
            {
                "role": "developer",
                "content": "Summarize this conversation excerpt in 2-3 sentences, preserving any facts, decisions, or commitments made. Be concise.",
            },
            {"role": "user", "content": transcript},
        ],
    )

    return {
        "role": "developer",
        "content": f"[Earlier conversation summary]: {summary_response.output_text}",
    }


def compact_history(history: list[dict], keep_recent: int = 10) -> list[dict]:
    system_message = history[0]
    rest = history[1:]

    if len(rest) <= keep_recent:
        return history  # nothing to compact yet

    to_summarize = rest[:-keep_recent]
    recent = rest[-keep_recent:]

    summary_message = summarize_old_messages(to_summarize)
    return [system_message, summary_message] + recent

Expected Behavior

Given a 60-message history, calling compact_history(history, keep_recent=10) replaces the oldest 50 messages with a single, compact summary message, keeping the system prompt and the 10 most recent messages verbatim. The resulting history is dramatically shorter in token count, while still giving the model a compressed sense of what happened earlier in the conversation — a meaningful improvement over Strategy 1's simple truncation, at the cost of an extra API call (to generate the summary) and the summary being, by definition, a lossy compression of the original exchange.

When to Use Summarization Over Truncation

Reach for summarization when a conversation's early content genuinely matters to later turns but doesn't need to be preserved verbatim — a long planning conversation where the user's stated goals and constraints from early on still matter, but the exact back-and-forth phrasing doesn't. Reach for simple truncation when old context genuinely stops being relevant — a support conversation where each new question is largely self-contained, or a conversation naturally organized around a single current topic where older topics are unlikely to be referenced again.

Strategy 3: Server-Side Compaction

The Responses API offers a server-side alternative that handles this automatically: passing context_management with a compact_threshold on your calls. When the rendered token count crosses your configured threshold, the API runs compaction on the server side automatically, returning an encrypted compaction item that represents the condensed prior context — you don't need to write your own summarization logic or manage the trimming yourself.

response = client.responses.create(
    model="gpt-5.6-luna",
    input=history,
    context_management={
        "compact_threshold": 300_000,  # trigger compaction above this token count
    },
)

This server-side approach works with both manually managed history (Lesson 2's input list pattern) and previous_response_id chaining (Lesson 3) — the server-side compaction happens transparently either way, and the resulting compacted context becomes what future calls in the chain build on. There's also a standalone /responses/compact endpoint for explicitly compacting a full context window on demand, separate from waiting for an automatic threshold to trigger — useful if you want to proactively compact at a predictable point (say, at the end of a topic) rather than reactively, once a token threshold is crossed.

An important detail: once you receive compacted output from this mechanism, you should pass it into your next call as-is, without trying to prune or further edit it yourself — the compacted context is specifically designed to be the canonical next context window, and manually second-guessing what it kept or dropped defeats the purpose of letting the server handle this for you.

Choosing a Strategy

None of these is universally correct — the right choice depends on how much your specific feature's older context actually matters to later turns, and how much control you want over exactly what gets preserved versus discarded.

Applying This to the Lesson 5 Chatbot Project

Let's extend the PersistentChatbot class from Lesson 5 with a budget check and automatic truncation, so the chatbot degrades gracefully instead of eventually erroring out on a long-running named conversation:

import tiktoken

MAX_HISTORY_TOKENS = 100_000
KEEP_RECENT_MESSAGES = 30


class PersistentChatbot:
    # ... __init__, _load, _save as in Lesson 5 ...

    def _estimate_tokens(self) -> int:
        encoding = tiktoken.get_encoding("cl100k_base")
        return sum(len(encoding.encode(m["content"])) + 4 for m in self.history)

    def _trim_if_needed(self) -> None:
        if self._estimate_tokens() <= MAX_HISTORY_TOKENS:
            return

        system_message = self.history[0]
        rest = self.history[1:]
        trimmed = [system_message] + rest[-KEEP_RECENT_MESSAGES:]
        self.history = trimmed
        self._save()

    def send(self, user_text: str) -> str:
        self._trim_if_needed()

        candidate = self.history + [{"role": "user", "content": user_text}]
        response = client.responses.create(model="gpt-5.6-luna", input=candidate)

        self.history = candidate + [
            {"role": "assistant", "content": response.output_text}
        ]
        self._save()
        return response.output_text

Calling _trim_if_needed() at the start of every send() means the chatbot checks and trims before it would ever risk exceeding the limit, rather than reactively catching an error after the fact. For this project, simple truncation (Strategy 1) is a reasonable choice given its simplicity, though converting _trim_if_needed() to use the summarization approach (Strategy 2) instead would be a natural, valuable extension for a conversation where losing old context entirely is a real cost.

Real-World Example: A Long-Running Coaching Assistant

Consider a genuinely long-lived use case: a personal coaching or journaling assistant a user talks to regularly over months. Early conversations establish real, durable context — goals the user stated, constraints they mentioned, patterns the assistant has helped them notice — that stays relevant far longer than a typical support chat's history would.

For a feature like this, blind truncation (Strategy 1) is a poor fit — losing the user's stated goals from three weeks ago because the conversation has since grown past a message-count cutoff would be a real regression in the assistant's usefulness. This is exactly the situation summarization (Strategy 2) or server-side compaction (Strategy 3) is built for: periodically condensing older sessions into a compact running summary — "user is working toward a marketing certification, prefers direct feedback, mentioned burnout concerns in week 2" — that persists indefinitely at a small, bounded token cost, while the verbatim recent exchanges stay available for the specific back-and-forth currently in progress. A team building this kind of long-lived assistant would likely combine techniques from across this entire unit: the Conversations API (Lesson 4) for cross-session durability, and a scheduled or threshold-triggered summarization pass (this lesson) to keep the durable context bounded rather than growing without limit for months on end.

Reading Actual Usage From a Response

Beyond local estimation with tiktoken, every response object tells you exactly how many tokens it actually used, which is the authoritative source of truth for monitoring real usage rather than relying purely on estimates:

response = client.responses.create(
    model="gpt-5.6-luna",
    input=history,
)

print("Input tokens:", response.usage.input_tokens)
print("Output tokens:", response.usage.output_tokens)
print("Total tokens:", response.usage.total_tokens)

Logging response.usage on every call — even just to a simple log file or a metrics dashboard — gives you real, historical visibility into how your conversations' token consumption grows over time, which is genuinely useful for two separate reasons: catching a conversation that's growing unexpectedly large before a user hits an error, and tracking the real cost implications of Lesson 2 and Lesson 3's point about re-sent history compounding token usage across a long conversation. A simple practice worth adopting: log response.usage.input_tokens alongside a conversation or session identifier, and periodically review which conversations are consuming the most tokens — this often surfaces genuinely useful product insights (which features produce unusually long conversations, whether a particular system prompt is unexpectedly verbose) beyond just the cost-management angle.

Monitoring Token Growth in Production

For a production system, it's worth building lightweight, ongoing monitoring rather than only checking token budgets reactively inside a single request handler. A simple approach: track a rolling average and maximum of input_tokens per conversation across your user base, and alert when a meaningful fraction of active conversations are approaching your configured trimming threshold.

def log_usage_metrics(conversation_id: str, response) -> None:
    record_metric(
        conversation_id=conversation_id,
        input_tokens=response.usage.input_tokens,
        output_tokens=response.usage.output_tokens,
        timestamp=datetime.utcnow(),
    )

This kind of lightweight instrumentation pays for itself the first time a product change unexpectedly causes conversations to grow much longer than anticipated — a new feature that encourages longer back-and-forth exchanges, for instance — and you want to catch that shift in your metrics before it shows up as a spike in context-length errors or unexpected cost increases in your OpenAI billing dashboard. Treating token usage as a metric worth tracking over time, the same way you'd track request latency or error rates, is a habit worth building early rather than only reaching for token counting reactively once a specific conversation has already become a problem.

A Note on Prompt Caching and Trimming Interaction

It's worth understanding how token-management strategies interact with OpenAI's automatic prompt caching, since the two can pull in slightly different directions. Prompt caching gives a cost discount on repeated leading content across calls — your system message and the stable early portion of a conversation, if unchanged between calls, are eligible for this discount. Trimming or summarizing early history, by definition, changes that leading content, which means a trimming or compaction event effectively "resets" the cacheable prefix for that conversation going forward — the next few calls after a compaction won't benefit from caching against the pre-compaction content, since that content no longer exists in the request.

This isn't a reason to avoid trimming or compaction — the token savings from a shorter context almost always outweigh the lost caching discount on a conversation that's grown large enough to need compaction in the first place. But it's worth knowing this interaction exists, particularly if you're tuning compaction thresholds for cost optimization specifically: a very aggressive compaction threshold (compacting very frequently) can end up costing more than a moderate one, because it repeatedly discards cacheable prefixes before they've had a chance to accumulate much caching benefit. A threshold set to compact only occasionally, once a conversation has genuinely grown large, tends to balance these two cost factors better than compacting on a very short interval.

Frequently Asked Questions

Do different models have different context window sizes? Yes, and this varies meaningfully across the model lineup — a smaller, faster model may have a more limited context window than a larger, more capable one, and this changes over time as OpenAI releases new models. Always check current documentation for the specific model you're using rather than assuming a number from memory or an older tutorial, since this is exactly the kind of detail that changes across model releases.

Should I pick a model partly based on its context window size? For any feature where conversations tend to run long — a coaching assistant, an extended research or planning tool — context window size is a legitimate factor in model selection, not just raw capability or cost. A cheaper model with a smaller context window might force compaction far more often than a pricier model with a larger window, and depending on your use case, that tradeoff might not be worth the savings.

Is it better to trim aggressively and stay well under the limit, or use as much of the context window as possible? This depends on your priorities. Staying well under the limit with a smaller, well-managed context tends to produce faster responses and lower cost per call, at the risk of losing more old context than strictly necessary. Using more of the available window preserves more context but costs more per call and can, in some cases, see reduced instruction-adherence on content buried deep in a very long context (as Lesson 2 noted). There's no universally correct answer — test with your specific use case and real usage patterns.

Can I combine local tiktoken estimation with server-side compaction? Yes, and this is a sensible combination — use local estimation for a cheap, fast proactive check ("are we getting close?") without an extra API call, and let server-side compaction (context_management) handle the actual compaction work once a real threshold is crossed, rather than implementing your own summarization logic from scratch.

Common Mistakes

Not checking token budget until an API error forces the issue. Reactive error handling for context-length errors is a poor user experience — the request has already failed, and the user is staring at an error instead of a response. Proactive checking, as this lesson's has_room_for() function demonstrates, catches the problem before it becomes a failed request.

Summarizing too aggressively, too often. Running a summarization pass on every single turn is wasteful — it costs an extra API call every time and can compound information loss if summaries are repeatedly re-summarized. Trigger compaction at a sensible threshold (a token count, or a fixed number of turns), not on every message.

Forgetting that reasoning tokens count toward the budget too. For reasoning-capable models like gpt-6-astra, a token budget calculated only from visible message content can significantly underestimate real usage, since reasoning tokens are part of the same shared budget but aren't always directly visible in your own token-counting code.

Applying the same trimming threshold across wildly different use cases. A quick support widget and a long-running personal assistant have very different tolerances for losing old context. Tune your trimming strategy and thresholds to the specific feature, rather than copying one configuration across every conversational feature in an application.

Troubleshooting

Getting a context-length-exceeded error despite having trimming logic. Double check the trimming logic is actually being invoked before the API call that's failing, not after, and confirm your token-counting estimate isn't undercounting — remember to include tool definitions and any reasoning-token overhead in your budget calculation, not just message content.

Summaries seem to lose important details over time. If you're re-summarizing an already-summarized block repeatedly, information loss compounds — consider keeping a slightly larger "recent" window before summarization kicks in, or making your summarization prompt more explicit about preserving specific categories of detail (stated facts, commitments, constraints) rather than a generic "summarize this."

Server-side compaction (context_management) doesn't seem to trigger. Confirm your compact_threshold value is actually below your conversation's accumulated token count — a threshold set too high simply never activates. Also confirm you're passing context_management consistently across the calls in your chain, since inconsistent configuration between calls can produce confusing behavior.

Token estimates from tiktoken don't exactly match what the API reports. This is expected — tiktoken-based local estimation is an approximation, not an exact replica of the API's internal counting (which includes additional formatting overhead you can't fully replicate client-side). Use local estimates for proactive budget checks, and treat response.usage from an actual API call as the authoritative source for post-hoc accounting.

Best Practices

Build token counting and budget checking into your conversation-management code from the start, even for a feature where you don't expect long conversations initially — retrofitting this after users have already produced problematically long conversations is far more painful than building it in from day one. Choose your compaction strategy based on how much your specific feature's older context actually matters, rather than defaulting to the simplest option everywhere. Reserve real headroom for the model's output when calculating whether you have "room" for a new turn — don't calculate right up to the hard limit. And whichever strategy you choose, test it explicitly with a genuinely long, realistic conversation before shipping — exactly the kind of testing that's easy to skip when your own manual testing never produces a conversation long enough to hit the problem.

Bringing the Unit Together

Across these six lessons, you've gone from the foundational fact that the API remembers nothing on its own (Lesson 1), through three distinct techniques for giving it memory anyway — full manual control (Lesson 2), lightweight server-side chaining (Lesson 3), and durable, addressable conversations (Lesson 4) — into a complete, working project that applies all of it (Lesson 5), and finally to the problem every one of those techniques eventually runs into as a conversation grows (this lesson). Every conversational feature you build from here forward, in this course or elsewhere, is a variation on these same core ideas: decide what state needs to persist, choose the technique that matches how that state needs to be stored and accessed, and plan for what happens when that state inevitably grows larger than you first expected.