Why Prompt Debugging Is Different from Code Debugging

When a piece of ordinary code misbehaves, you can typically reproduce the failure deterministically, step through execution with a debugger, and inspect exact intermediate state at each line. A misbehaving prompt resists all three of these habits by default. The same input can, on a nonzero-temperature request, produce a different output on each retry, so "it worked when I just ran it" tells you very little about whether the underlying prompt is reliable. There is no line-by-line execution to step through — the model's internal computation is opaque, and the only artifacts you have to work with are the input you sent and the output you got back. And unlike a stack trace pointing at an exact failing line, a bad output rarely tells you which part of your instructions caused it, or whether the instructions were even the problem at all rather than the input data, the model choice, or a parameter setting.

This lesson treats prompt debugging as its own discipline, with its own systematic process, rather than an ad hoc "tweak the wording and try again" activity. The process draws directly on every technique this unit has already covered — the instructions/input distinction (Lesson 1), precise instruction-writing (Lesson 2), few-shot examples (Lesson 3), and reasoning effort (Lesson 4) — and applies them as diagnostic tools rather than only as prompt-writing tools.

Step 1: Establish Whether the Failure Is Reproducible

Before changing anything, determine whether the failure happens every time with the same input, or only sometimes. This single distinction determines almost everything about how to proceed, so it's worth confirming explicitly rather than assuming.

def check_reproducibility(instructions: str, input_text: str, model: str = "gpt-5.6-luna",
                            n_trials: int = 10, temperature: float = 1.0) -> dict:
    """Run the same request multiple times and report how consistent the outputs are."""
    outputs = []
    for _ in range(n_trials):
        response = client.responses.create(
            model=model,
            instructions=instructions,
            input=input_text,
            temperature=temperature,
        )
        outputs.append(response.output_text.strip())

    unique_outputs = set(outputs)
    return {
        "n_trials": n_trials,
        "n_unique_outputs": len(unique_outputs),
        "outputs": outputs,
        "fully_consistent": len(unique_outputs) == 1,
    }

result = check_reproducibility(
    instructions="Extract the shipping deadline from the message. Respond with just the date.",
    input_text="We really need this by the fifteenth, or the following Monday at the latest if that's not doable.",
)
print(f"Unique outputs across {result['n_trials']} trials: {result['n_unique_outputs']}")
for o in result["outputs"]:
    print(f"  - {o}")

A failure that reproduces on every trial, even at temperature=0, is a systematic problem — the instructions genuinely don't tell the model what you want, or the input is genuinely ambiguous in a way no reasonable interpretation resolves. A failure that appears on only some trials at nonzero temperature is a consistency problem of the kind Lesson 2 addressed — the instruction leaves room for legitimate variation, and the fix is more likely to be tightening ambiguity or lowering temperature than a wholesale rewrite. Treating these as the same kind of problem, and reaching for the same fix, is one of the most common inefficiencies in prompt debugging — a developer who rewrites an entire instruction block in response to one bad output that turns out to have been a rare, temperature-driven fluke has wasted effort solving a problem that a temperature=0 retest would have shown didn't reliably exist.

Step 2: Isolate Instructions from Input as Separate Suspects

Because instructions and input serve different roles (Lesson 1), a misbehaving response can originate from either one, and conflating them slows down debugging. A practical isolation technique is to hold one constant while varying the other, systematically, rather than changing both at once.

def isolate_failure_source(instructions: str, input_variants: list[str], model: str = "gpt-5.6-luna"):
    """Hold instructions fixed, vary only the input, to see if the problem tracks the input."""
    print("Testing with fixed instructions, varying input:")
    for variant in input_variants:
        response = client.responses.create(model=model, instructions=instructions, input=variant)
        print(f"  Input: {variant!r}")
        print(f"  Output: {response.output_text.strip()!r}\n")

instructions = "Extract the shipping deadline from the message. Respond with just the date, in YYYY-MM-DD format. Assume the current year is 2026."
isolate_failure_source(instructions, [
    "We need this by March 15th.",
    "We need this by the fifteenth of next month.",
    "We need this ASAP, ideally by next Friday.",
])

If the output is wrong or inconsistent for every variant, the problem most likely lies in the instructions — the task description itself is incomplete or ambiguous regardless of what specific input it's applied to (in the example above, "next Friday" and "next month" both require the model to know today's date, which the instructions never supply — a genuine gap, not a fluke). If the output is correct for most variants but wrong for one specific kind of input (dates expressed relative to "today," say), the problem is more likely a case your instructions don't yet cover, meaning the fix is adding a rule or example addressing that specific case rather than rewriting the whole instruction. This distinction directly determines which of Lesson 2's or Lesson 3's techniques to reach for: a systemic ambiguity calls for tightening the verbal instruction, while a single uncovered edge case is often best fixed with one additional few-shot example demonstrating exactly that case.

Step 3: Check Whether the Model Itself Is the Right Choice

Before extensively rewriting a prompt, it's worth ruling out a simpler cause: the request may be using a model that isn't well suited to the task's actual difficulty. Unit 2, Lesson 5 covered choosing a model for a task; a prompt that seems to misbehave no matter how it's rewritten is sometimes actually being asked too much of the model it's running on, particularly if the task involves multi-step reasoning and the current model has no reasoning capability at all.

def compare_across_models(instructions: str, input_text: str, models: list[str]):
    """Run the identical prompt against multiple models to see if the failure is model-specific."""
    for model in models:
        kwargs = {"model": model, "instructions": instructions, "input": input_text}
        if model == "gpt-6-astra":
            kwargs["reasoning"] = {"effort": "medium"}
        response = client.responses.create(**kwargs)
        print(f"{model}: {response.output_text.strip()}")

compare_across_models(
    instructions="Given the constraints below, determine the optimal delivery route. Explain your reasoning briefly.",
    input_text="Deliver to Warehouse A, B, and C. A must precede C. Total driving time must stay under 5 hours. ...",
    models=["gpt-5.6-luna", "gpt-5.6-terra", "gpt-6-astra"],
)

If a cheaper, non-reasoning model consistently fails a task that a reasoning model consistently handles correctly, the original prompt may never have been the problem — the task genuinely required more deliberation than the chosen model provides, and Lesson 4's guidance about matching model and effort level to task difficulty applies directly. This check is worth running early in a debugging session, because it can save considerable time that would otherwise go into rewriting instructions that were never the actual cause of the failure.

Step 4: Read the Instructions as the Model Would, Not as You Meant Them

One of the most persistently useful debugging techniques has nothing to do with code at all: read your own instructions cold, as if you had never written them and had no access to the intention behind them, looking specifically for anything that could be reasonably interpreted more than one way. This is difficult to do with instructions you just wrote, because you already know what you meant — the fix is to introduce distance, either by setting the instructions aside and returning after a break, or by asking a colleague (or another model, as a genuinely useful debugging trick) to state back what they think the instructions require.

def get_interpretation(instructions: str) -> str:
    """Ask a model to restate what it understands the instructions to require — a
    useful way to surface ambiguity you can no longer see because you wrote the prompt."""
    response = client.responses.create(
        model="gpt-5.6-luna",
        instructions="Restate, in your own words, exactly what the following instructions "
                     "require you to do. Be specific about anything that seems ambiguous "
                     "or underspecified.",
        input=instructions,
    )
    return response.output_text

instructions_to_check = "Summarize the ticket and flag if it's urgent."
print(get_interpretation(instructions_to_check))

A model asked to restate this instruction will often surface exactly the kind of ambiguity Lesson 2 warned about — what counts as "urgent" is never defined, and "flag" doesn't specify where or how the flag should appear in the output. Seeing this stated back explicitly, by a system with no access to what you actually intended, is often more revealing than staring at your own instructions directly, precisely because it removes the unconscious mental patching a prompt's author does automatically when reading their own words.

Step 5: Test the Boundary Cases Deliberately, Not Only the Common Case

A prompt that works well on typical, central examples of the task can still fail regularly on boundary cases — inputs that sit right at the edge between two categories, or that combine several complicating factors your central test cases didn't include. Debugging should include deliberately constructing these boundary cases rather than only re-testing the cases that already work.

BOUNDARY_TEST_CASES = [
    {"input": "", "note": "empty input"},
    {"input": "N/A", "note": "explicitly non-informative input"},
    {"input": "Maybe by the 15th? Not totally sure yet.", "note": "uncertain, hedged date"},
    {"input": "By the 15th, or actually let's say the 20th instead.", "note": "self-correcting input"},
    {"input": "Deadline: " + "urgent " * 200, "note": "unusually long, repetitive input"},
]

def test_boundaries(instructions: str, cases: list[dict], model: str = "gpt-5.6-luna"):
    for case in cases:
        response = client.responses.create(model=model, instructions=instructions, input=case["input"])
        print(f"[{case['note']}] -> {response.output_text.strip()!r}")

test_boundaries(
    instructions="Extract the shipping deadline. Respond with just the date, or 'none' if no date is given.",
    cases=BOUNDARY_TEST_CASES,
)

Boundary cases like these routinely surface failures that never show up in ordinary testing: an empty input might cause the model to fabricate a plausible-sounding date rather than correctly responding "none"; a self-correcting input ("the 15th, or actually the 20th") tests whether the instruction handles the last stated value correctly rather than the first; and an unusually long or repetitive input can sometimes cause a model to lose track of the actual task amid the repeated text. None of these are exotic scenarios — they are the ordinary messiness of real user input, and a prompt that has not been tested against them is not yet ready for production traffic, however well it performs on the clean cases used during initial development.

Step 6: Change One Variable at a Time

When a debugging session does call for a change — a tightened instruction, an added few-shot example, an adjusted effort level, a different model — resist making several changes simultaneously before retesting. Changing the instruction wording, adding an example, and switching models all at once, then observing that the output improved, tells you that something in that combination helped, but not which change actually mattered, which means you cannot confidently keep only the useful part and discard the parts that added complexity or cost without benefit.

def ab_test_change(base_config: dict, changed_config: dict, test_inputs: list[str]) -> dict:
    """Compare exactly two configurations that differ in one respect, across several inputs."""
    results = {"base": [], "changed": []}
    for label, config in [("base", base_config), ("changed", changed_config)]:
        for text in test_inputs:
            response = client.responses.create(input=text, **config)
            results[label].append(response.output_text.strip())
    return results

base = {"model": "gpt-5.6-luna", "instructions": "Extract the deadline. Respond with just the date."}
changed = {"model": "gpt-5.6-luna",
           "instructions": "Extract the deadline. Respond with just the date in YYYY-MM-DD format, "
                           "or 'none' if no date is mentioned. Assume the current year is 2026."}

outcomes = ab_test_change(base, changed, [
    "Ship by March 3rd.",
    "No rush on this one.",
    "Need it by the end of next month.",
])
for label in ("base", "changed"):
    print(f"{label}: {outcomes[label]}")

This discipline is the same principle behind Lesson 2's recommendation to keep a small suite of test cases and re-run them after every prompt change, applied specifically to the debugging process: one change, one retest, one conclusion, before moving to the next change. It is slower per iteration than changing several things at once, but it produces a debugging history you can actually trust and learn from, rather than a final working prompt whose individual pieces you no longer understand the purpose of.

Failure Signatures: Matching Symptoms to Likely Causes

Experienced prompt debugging benefits from recognizing a handful of recurring failure "signatures" — the shape a bad output takes often points toward its likely cause before you've run a single diagnostic step, in the same way a stack trace's exception type narrows down where to look in ordinary code.

This table is a starting point for triage, not a substitute for actually running the diagnostic steps above — a given symptom can have more than one plausible cause, and the table's purpose is to help you choose which diagnostic step to try first rather than to replace the diagnostic process itself.

Step 7: Read the Token-Level Evidence, Not Just the Final Text

Beyond the visible output text, every response carries usage details that are themselves useful debugging evidence, particularly for the truncation and reasoning-cost failure modes introduced in Lesson 4. Get in the habit of inspecting usage whenever an output looks wrong, not only when you suspect a cost problem.

def diagnose_response(response) -> None:
    """Print a quick diagnostic summary of a response's usage, useful whenever the
    visible output looks wrong and the cause isn't immediately obvious."""
    usage = response.usage
    print(f"Output text length: {len(response.output_text)} chars")
    print(f"Input tokens: {usage.input_tokens} (cached: {usage.input_tokens_details.cached_tokens})")
    print(f"Output tokens: {usage.output_tokens}")
    reasoning = getattr(usage.output_tokens_details, "reasoning_tokens", 0)
    if reasoning:
        print(f"  of which reasoning tokens: {reasoning}")
        visible = usage.output_tokens - reasoning
        print(f"  of which visible answer tokens: {visible}")
        if visible == 0:
            print("  WARNING: entire output budget was consumed by reasoning — likely truncated.")

response = client.responses.create(
    model="gpt-6-astra",
    input="A very hard multi-step logic puzzle...",
    reasoning={"effort": "high"},
    max_output_tokens=300,
)
diagnose_response(response)

An empty visible answer alongside a reasoning_tokens count near the max_output_tokens ceiling is close to a direct confirmation of the truncation failure mode from Lesson 4 — evidence you would miss entirely if you only looked at the empty string returned and assumed, incorrectly, that the model simply "failed" or that the prompt itself was somehow at fault. Similarly, a surprisingly low cached_tokens value on a request whose instructions you believed to be a stable, cacheable prefix can reveal that the instructions are unintentionally changing slightly between calls — perhaps a timestamp or a request ID is being interpolated directly into the instructions string rather than kept in input where it belongs, which breaks the caching benefit described in Unit 1 and Lesson 3 without any visible symptom in the output text itself.

A Full Worked Debugging Session

To see the process applied end to end, consider a customer-feedback classifier that a team reports is "inconsistent in production" — vague enough that it could be almost any of the failure modes above.

Reported symptom: the classifier sometimes labels clearly negative feedback as "neutral."

Step 1 — reproducibility check. Running the exact reported input through check_reproducibility() at temperature=0 ten times produces the same "neutral" label every time. This immediately rules out a temperature-driven fluke — the failure is systematic, not random, which means the fix belongs in the instructions or examples, not in a temperature or seed adjustment.

Step 2 — instruction/input isolation. Testing the same instructions against several other clearly negative inputs shows the same "neutral" mislabeling occurs specifically on feedback that is negative but phrased politely — "The response time could have been a lot better, unfortunately" gets labeled "neutral," while "This is terrible, completely unacceptable" correctly gets labeled "negative." The failure tracks a specific style of input, not the instructions failing universally.

Step 4 — restate-back test. Asking a model to restate the classifier's instructions reveals the actual gap: the instructions define "negative" only by example of harshly-worded complaints, with no mention that politely-worded critical feedback is still negative in substance. The instructions were never wrong about the label set — they simply never addressed the tone-versus-substance distinction that turns out to matter for this specific business's feedback data.

Fix — targeted near-miss example (Lesson 3). Rather than rewriting the whole instruction, one near-miss example is added, pairing polite phrasing with the correct negative label:

instructions = """Classify feedback sentiment as positive, neutral, or negative, based on
the substance of the feedback, not how politely it is phrased.

Example:
Feedback: "The response time could have been a lot better, unfortunately."
Sentiment: negative

Example:
Feedback: "This is terrible, completely unacceptable."
Sentiment: negative"""

Re-test. Running the original boundary-case set (Step 5) that included several politely-worded negative examples now produces correct "negative" labels across the board, and the reproducibility check confirms the fix holds consistently across ten trials at temperature=0. The team's vague "inconsistent" report turned out to be a specific, fixable gap — the instructions never distinguished tone from substance — that a full rewrite would have addressed only by accident, whereas the systematic process isolated it directly and fixed it with a single added example.

This worked example illustrates why the ordered process in this lesson matters more than any single technique in isolation: a debugging session that jumped straight to "add more examples" or "try a different model" without first confirming reproducibility and isolating the failure to a specific input style could easily have spent considerably more effort arriving at a similar fix, or worse, arrived at a different fix that patched the reported symptom without addressing its actual cause.

Common Mistakes

Concluding a prompt is broken from a single bad output. A single failure, especially at nonzero temperature, tells you nothing about whether the prompt is systematically unreliable or whether you happened to see a rare unlucky sample; always check reproducibility (Step 1) before treating one bad output as proof of a systemic problem.

Rewriting the entire prompt in response to one narrow failure. A large rewrite makes it impossible to know afterward which part of the change actually fixed the problem, and risks introducing new issues elsewhere in the prompt that a narrower, targeted fix would have avoided.

Debugging by only retesting the cases that already work. This confirms that a fix didn't break what was already working, but it tells you nothing about whether it fixed the actual failure or addressed the boundary cases most likely to fail in production.

Assuming the problem is always the prompt, never the model choice. A task that is genuinely too difficult for a chosen model's reasoning capability will not be reliably fixed by prompt wording alone, however carefully it's rewritten; Step 3's model comparison check exists specifically to rule this out early.

Changing multiple variables in the same debugging iteration. This trades diagnostic clarity for iteration speed, and the trade is rarely worth it — a debugging process that can't tell you which specific change fixed a problem will struggle to prevent that same problem from resurfacing later.

Best Practices

Keep a running log of prompt versions alongside their test results, not just the current version — when a later change unexpectedly regresses behavior that used to work, a version history lets you identify exactly which change introduced the regression rather than re-deriving the fix from scratch.

Build your boundary test cases once and reuse them across every future revision of the same prompt, exactly as Lesson 2 recommended for consistency testing generally — a boundary case that broke a prompt once is exactly the kind of case likely to break a future revision too, if it's not deliberately re-tested.

Treat "restate the instructions back to me" (Step 4) as a routine step before shipping any new prompt, not only as a last resort during active debugging — catching ambiguity before deployment is considerably cheaper than diagnosing it afterward from a stream of inconsistent production outputs.

Separate the question "is this reproducible" from the question "is this correct." A highly reproducible, consistent output that is confidently wrong every time is arguably a worse production risk than an inconsistent one, because its wrongness may go unnoticed longer; both are real bugs, but they call for different diagnostic paths — Step 2's instruction/input isolation for the reproducible case, and Lesson 2's consistency techniques for the inconsistent one.

Escalate to systematic evaluation once ad hoc debugging stops scaling. The techniques in this lesson work well for diagnosing an individual misbehaving prompt, but a production application with many prompts and continuous traffic needs the more structured evaluation framework covered in Unit 13 — treat this lesson's process as the foundation that framework builds on, not a replacement for it.

Turning Fixed Bugs into a Regression Suite

Every bug this process finds and fixes is worth preserving as a permanent test case, not just resolving and moving on from. A lightweight regression suite — a plain list of input/expected-output pairs, each one originally a real failure — turns individual debugging sessions into a cumulative safety net that protects future prompt changes from reintroducing problems you have already solved once.

REGRESSION_CASES = [
    {"input": "The response time could have been a lot better, unfortunately.",
     "expected": "negative", "note": "politely-worded negative feedback (fixed 2026-09)"},
    {"input": "", "expected": "none", "note": "empty input should not produce a fabricated date"},
    {"input": "By the 15th, or actually let's say the 20th instead.",
     "expected": "2026-09-20", "note": "self-correcting input; last stated date wins"},
]

def run_regression_suite(instructions: str, cases: list[dict], model: str = "gpt-5.6-luna") -> bool:
    all_passed = True
    for case in cases:
        response = client.responses.create(model=model, instructions=instructions, input=case["input"])
        actual = response.output_text.strip()
        passed = actual == case["expected"]
        all_passed = all_passed and passed
        status = "PASS" if passed else "FAIL"
        print(f"[{status}] {case['note']}: expected {case['expected']!r}, got {actual!r}")
    return all_passed

run_regression_suite(instructions, REGRESSION_CASES)

Running this suite before shipping any future change to the same prompt — exactly the discipline Lesson 2 recommended for consistency testing generally — converts what would otherwise be tribal knowledge ("we fixed a bug like this once, a while back") into an automated check that catches a regression immediately, the moment a future edit reintroduces a previously-fixed failure mode, rather than weeks later when the same complaint resurfaces from production traffic.

Treat the regression suite itself as a living artifact that grows over the lifetime of a prompt, not a one-time deliverable produced during initial development. Every subsequent debugging session that this lesson's process resolves should end with one more entry added to the suite, and the suite should be run — automatically, as part of whatever deployment process ships a prompt change — before any revised instructions string reaches production traffic. A team that maintains this discipline consistently will find that the categories of bugs recurring in later debugging sessions shift over time, away from the boundary cases and ambiguities the suite already guards against and toward genuinely new failure modes surfaced by evolving input data or a model version change — which is itself a useful signal that the debugging process described in this lesson is doing its job.