Prompt and context optimization for lower cost

Unit 12, Lesson 4 covered prompt-prefix caching, which reduces the cost of resending an unchanged prefix across calls. This lesson covers a different, complementary technique: reducing the actual number of tokens in the prompt itself, independent of whether caching is available. A shorter prompt is cheaper every time, cached or not.

Why Trimming Is Different From Caching

Prompt caching reduces the cost of tokens you send repeatedly by discounting the ones that were already seen. It does nothing for the very first call, and it does nothing for the portion of the prompt that changes on every call — typically the user's actual message and any freshly retrieved context. Trimming attacks a different lever: it reduces the raw number of tokens that need to be sent and processed at all, which lowers cost regardless of caching state and also reduces latency, since fewer input tokens generally means a faster time-to-first-token.

The two techniques stack. A well-trimmed prompt with a stable, cacheable prefix gets the benefit of both: less to send in the first place, and a discount on the part that repeats.

Identifying What Is Actually Necessary

The starting point for trimming is auditing what is currently in the prompt and asking, for each piece, whether the model's output would change if it were removed. Common sources of unnecessary bulk include:

• Verbose instructions that repeat themselves. A system prompt that explains the same constraint three different ways in three different paragraphs costs three times the tokens for the same effective instruction.
• Full conversation history when only recent turns matter. Many tasks do not need the entire conversation resent on every turn — a summarized version of older turns can preserve necessary context at a fraction of the token cost.
• Retrieved documents included in full when only a section is relevant. Retrieval-augmented generation systems often over-retrieve, including entire documents when a single paragraph would answer the question.
• Redundant formatting instructions restated per call. If the response format never changes, it belongs once in a stable system prompt rather than repeated in every user message.

Trimming Conversation History

A straightforward and high-impact technique for long-running conversations is capping how much history is resent, replacing older turns with a compact summary rather than dropping them or sending them in full.

def build_trimmed_history(
    full_history: list[dict],
    max_recent_turns: int = 6,
    summary: str | None = None,
) -> list[dict]:
    recent = full_history[-max_recent_turns:]
    if summary is None or len(full_history) <= max_recent_turns:
        return recent

    summary_message = {
        "role": "system",
        "content": f"Summary of earlier conversation: {summary}",
    }
    return [summary_message] + recent

This function keeps only the most recent max_recent_turns messages verbatim, since recent turns are almost always the most relevant to the current request, and replaces everything older with a single compact summary_message. The token cost of the summary is typically a small fraction of the token cost of the full history it replaces, since a summary compresses many turns into a few sentences. The if summary is None or len(full_history) <= max_recent_turns guard avoids inserting an unnecessary summary message when the conversation is already short enough that no trimming was needed, which would otherwise add tokens for no benefit.

Producing the summary itself typically requires a separate, infrequent model call (updated periodically as the conversation grows, not on every single turn) — the cost of periodically summarizing is usually far lower than the cost of resending an ever-growing history on every turn indefinitely.

Trimming Retrieved Context

In a retrieval-augmented system, the retrieval step often returns more content than the query actually needs — a full document when one paragraph is relevant, or several candidate chunks when only the top one or two are useful.

def select_relevant_chunks(
    chunks: list[tuple[str, float]],  # (chunk_text, relevance_score)
    max_chunks: int = 3,
    min_score: float = 0.5,
) -> list[str]:
    filtered = [text for text, score in chunks if score >= min_score]
    return filtered[:max_chunks]


def build_context_block(chunks: list[str]) -> str:
    if not chunks:
        return ""
    numbered = [f"[{i + 1}] {chunk}" for i, chunk in enumerate(chunks)]
    return "Relevant context:\n" + "\n".join(numbered)

select_relevant_chunks applies two independent limits: a min_score threshold that drops chunks the retrieval system itself considered weakly relevant, and a max_chunks cap that bounds the total context size regardless of how many chunks pass the threshold. Both matter — the score threshold alone does not prevent a query that happens to match many chunks moderately well from including all of them, while the count cap alone does not prevent weak, irrelevant matches from taking a slot that a strong match should occupy. Together they bound both the quality and quantity of injected context, which directly bounds the token cost of the retrieval portion of the prompt.

Numbering the chunks in build_context_block ([1], [2], [3]) is a small but meaningful detail: it lets the model's response reference a specific source (“as shown in [2]”), which is often required for citation-style features, without materially increasing token count.

Compressing Instructions Without Losing Meaning

System prompts often accumulate redundant phrasing over iterative development — a constraint added, then re-stated slightly differently when a related edge case comes up, without removing the original.

verbose_system_prompt = """
You are a helpful assistant. Please always be polite and respectful.
Make sure to use a friendly tone at all times. Do not be rude or dismissive.
Always respond politely, even if the user is frustrated. Keep responses short.
Try to keep your answers brief and to the point. Avoid long responses.
"""

concise_system_prompt = """
You are a helpful, polite assistant. Keep responses brief.
"""

The verbose version restates "be polite" three times and "be brief" twice, in different words, without adding any new constraint the model does not already receive from the first mention. Consolidating repeated instructions into a single clear statement per constraint reduces token count substantially for system prompts that are sent on every single call, which compounds significantly at scale — a system prompt sent on every one of a million daily requests pays its token cost a million times over, so even a modest per-call reduction produces a large aggregate savings.

Note: This does not mean shorter is always better for quality — some instructions genuinely benefit from restatement or examples for reliability, particularly for complex formatting requirements. The goal is removing redundant restatement, not removing necessary detail. Always validate that quality holds after trimming, not just that token count dropped.

Measuring the Impact of Trimming

Any trimming change should be measured, not assumed. Combining the token counting utility from Lesson 2 with a simple before/after comparison quantifies the actual savings.

def compare_prompt_token_cost(
    original_messages: list[dict],
    trimmed_messages: list[dict],
    model: str = "gpt-5.6-terra",
) -> dict:
    def total_tokens(messages: list[dict]) -> int:
        return sum(count_tokens(m["content"], model) for m in messages)

    original_tokens = total_tokens(original_messages)
    trimmed_tokens = total_tokens(trimmed_messages)
    reduction_pct = (
        ((original_tokens - trimmed_tokens) / original_tokens) * 100
        if original_tokens > 0
        else 0.0
    )

    return {
        "original_tokens": original_tokens,
        "trimmed_tokens": trimmed_tokens,
        "reduction_pct": round(reduction_pct, 1),
    }

This reuses count_tokens from Lesson 2 rather than reimplementing token counting, keeping the token-counting logic in one place. Reporting reduction_pct alongside the raw counts makes the impact of a trimming change immediately interpretable in a code review or a changelog ("this reduces prompt tokens by 34%") rather than requiring the reader to do the arithmetic themselves.

Testing Trimming Logic

Trimming functions are pure data transformations and should be tested with constructed fixtures, independent of any real tokenizer or model call.

def test_build_trimmed_history_keeps_recent_turns_and_summary():
    history = [{"role": "user", "content": f"message {i}"} for i in range(10)]
    trimmed = build_trimmed_history(history, max_recent_turns=3, summary="earlier discussion about pricing")

    assert len(trimmed) == 4  # 1 summary message + 3 recent turns
    assert trimmed[0]["role"] == "system"
    assert "pricing" in trimmed[0]["content"]
    assert trimmed[-1]["content"] == "message 9"
    print("PASS: trimmed history keeps recent turns and prepends summary")


def test_select_relevant_chunks_applies_both_limits():
    chunks = [
        ("strong match", 0.9),
        ("medium match", 0.6),
        ("weak match", 0.3),
        ("another strong match", 0.85),
    ]
    selected = select_relevant_chunks(chunks, max_chunks=2, min_score=0.5)
    assert selected == ["strong match", "medium match"]
    print("PASS: chunk selection respects both score threshold and count cap")


test_build_trimmed_history_keeps_recent_turns_and_summary()
test_select_relevant_chunks_applies_both_limits()

The first test checks both the count (len(trimmed) == 4) and the content placement (summary first, most recent message last) — a trimming function that returned the right number of messages in the wrong order would still pass a count-only test, so verifying order matters. The second test uses chunks crafted to have distinct scores specifically so the min_score filter and the max_chunks cap each visibly affect the result, confirming neither limit is silently ignored.

Common Mistakes

Trimming context so aggressively that answer quality degrades. Removing tokens that the model actually needed to answer correctly trades a small cost saving for a much larger cost: a wrong or incomplete answer that requires a retry, or worse, an undetected quality regression. Always evaluate output quality after a trimming change, not just token count.

Resending full conversation history indefinitely without any cap. In a long-running conversation, unbounded history growth means every single turn resends every previous turn, so cost per turn grows linearly with conversation length even though most of that history is no longer relevant to the current question.

Treating prompt trimming and prompt caching as interchangeable. They solve different problems and should both be applied where appropriate — trimming reduces raw token count, caching reduces the cost of resending tokens that do not change. Skipping trimming because caching is already in place leaves savings on the table for the ever-changing portion of the prompt.

Best Practices

Audit prompts periodically, not just once. System prompts accumulate redundant instructions over time as new edge cases prompt new additions; a periodic review to consolidate and remove overlap keeps token count from creeping upward indefinitely.

Cap and summarize rather than truncate blindly. Simply cutting off old conversation turns loses information the user may reference later; summarizing preserves the gist at a fraction of the token cost, which is almost always the better trade-off.

Quantify every trimming change with a before/after comparison. Use a function like compare_prompt_token_cost to make the savings from a specific change visible and reviewable, rather than relying on an impression that a prompt "feels shorter."