Debugging a Multi-Step Prompt Workflow

Why Debugging Chains Is Different When a single prompt gives a wrong answer, there is one place to look. When a five-step chain gives a wrong answer, the cause could be in any step, in the handoff between steps, in a validation rule that let a bad value through, in code that made a decision, or in the input itself. And because models can vary between runs, the error may not even appear every time.

Debugging a chain therefore needs a method, not trial and error:

1. Reproduce the failure ↓
2. Inspect the trace ↓
3. Find the first step whose output is wrong ↓
4. Classify the root cause ↓
5. Fix it at the right layer ↓
6. Add a regression test so it cannot return unnoticed This lesson walks through each step, with code for inspecting traces, locating the failing step, and replaying a single step in isolation.

Step 1: Reproduce the Failure You cannot fix what you cannot observe. Collect:

The exact input that produced the wrong result. The prompt versions and model used (Lesson 67). The trace of that run (Lesson 62), if it was recorded. Then run the chain again on the same input.

If the Failure Does Not Reproduce Model outputs vary between runs (Lesson 2). If the failure appears only sometimes:

Run the input several times, for example 10, and count how often it fails. A failure that appears in 3 of 10 runs is a reliability problem: the prompt leaves room for different interpretations, or a borderline case is not decided by a clear rule (Lesson 24). Treat intermittent failures as real. A failure rate of 30% on one input type in testing can become thousands of failures in production. Step 2: Inspect the Trace A trace records each step's input and output (Lesson 62). Without it, debugging requires guessing. With it, you can read the chain's work step by step.

A Trace for a Wrong Result A customer bought a course 10 days ago and asked for a refund. The policy allows refunds only within 7 days, but the chain sent a reply saying the refund was approved.

trace = [ {"step": "extract", "input": {"ticket": "Bought Django 10 days ago, want refund"}, "output": {"issue_type": "refund_request", "days_since_purchase": 10}}, {"step": "decision", "input": {"days_since_purchase": 10}, "output": "refund_eligible"}, {"step": "draft", "input": {"decision": "refund_eligible"}, "output": "Good news! Your refund has been approved."}, ] Reading the Trace extract: the issue type and 10 days are correct. decision: the output is refund_eligible, but 10 days is beyond the 7-day limit. This is wrong. draft: the reply is a correct reply for the decision it received. The draft step did its job; it was given a wrong decision. The final output was wrong, but the draft step was not the cause. This is the most important principle of chain debugging:

The step where an error becomes visible is often not the step where it started.

Step 3: Find the First Step Whose Output Is Wrong For each step, decide what a correct output would be for this input, and check the steps in order. The first step with a wrong output is where the problem started; later steps may only be following it.

Doing This in Code When you know what each step should have produced for a specific input, you can write expectations and let code find the first failure.

import json

expectations = { "extract": lambda out: out.get("issue_type") == "refund_request" and out.get("days_since_purchase") == 10, "decision": lambda out: out == "refund_not_eligible", "draft": lambda out: "approved" not in out.lower(), }

def first_failing_step(trace: list[dict], expectations: dict) -> dict | None: """Return the first trace record whose output does not meet its expectation.""" for record in trace: check = expectations.get(record["step"]) if check and not check(record["output"]): return record return None

failing = first_failing_step(trace, expectations) print("First failing step:", failing["step"]) print("Its input:", json.dumps(failing["input"])) print("Its output:", failing["output"]) Output:

First failing step: decision Its input: {"days_since_purchase": 10} Its output: refund_eligible Code Explanation expectations A dictionary of small functions, one per step, each returning True if the output is correct for this test case. They express what you, the developer, know the correct intermediate results should be.

first_failing_step() Walks the trace in order and returns the first record that fails its expectation. Both the draft step and the decision step fail their checks here, but the decision step comes first, so it is reported.

Why "first" matters If you fixed the draft step, for example by telling it never to approve refunds, you would hide the symptom while leaving the real bug in place. The wrong decision would still affect other replies, logs, and reports.

When Expectations Are Not Known For complex outputs, you may not have exact expectations. Read each step's input and output and ask: "Given this input, is this output correct?" If yes, the step is fine, even if the final result is wrong. Move to the next step.

Step 4: Classify the Root Cause Once you know where the error started, determine why. Most chain errors fall into a few categories.

Root Cause Symptoms Example Prompt problem The step received correct input but produced a wrong output An extraction prompt does not say how to handle "two weeks ago" Input problem The input itself is unusual, ambiguous, or malformed A ticket in mixed Tamil and English that the prompt does not anticipate Handoff problem The step received wrong, incomplete, or badly formatted input from the previous step The decision step received days as a string "10" instead of the integer 10 Validation gap A wrong output was not caught between steps No check that days_since_purchase is an integer Code logic problem A code step implemented a rule incorrectly The refund rule uses < 14 instead of <= 7 Model variance The step fails only on some runs Borderline inputs decided differently each time Changed environment The chain worked before and fails now Model updated, policy changed, prompt edited (Lesson 67) Replaying a Single Step To test a hypothesis about one step, run only that step with the exact input recorded in the trace. This is called replaying the step. It isolates the step from the rest of the chain and from earlier randomness.

def decide(state: dict) -> str: """The current (buggy) rule.""" return "refund_eligible" if state["days_since_purchase"] < 14 else "refund_not_eligible"

def decide_fixed(state: dict) -> str: """The corrected rule: refunds within 7 days of purchase.""" return "refund_eligible" if state["days_since_purchase"] <= 7 else "refund_not_eligible"

def replay(step_function, record: dict): """Run one step again with the input recorded in the trace.""" return step_function(record["input"])

print("Replay buggy:", replay(decide, failing), " Replay fixed:", replay(decide_fixed, failing)) Output:

Replay buggy: refund_eligible Replay fixed: refund_not_eligible What the Replay Shows The decision step received the correct input (10 days) and produced the wrong output. The cause is a code logic problem: the rule used 14 days instead of 7. The replay confirms both the bug and the fix, without running any model calls.

For a model step, replaying means sending the recorded input to the step's prompt again, several times if necessary, to see whether the failure is consistent (a prompt problem) or intermittent (variance).

Step 5: Fix It at the Right Layer Fix the cause where it lives, not where it becomes visible.

Root Cause Right Fix Wrong Fix Prompt problem Clarify the step's prompt; add a rule or example (Lesson 25) Patch a later step to compensate Input problem Add handling for this input type to the prompt or preprocessing Ignore it because it is "rare" Handoff problem Fix the output format or the selection of state (Lesson 71) Make the next step guess the missing information Validation gap Add a check after the step (Lesson 73) Rely on the final output check Code logic problem Fix the code and add a unit test Add instructions to a prompt to correct the code's result Model variance Add a boundary rule, examples, or self-consistency (Lessons 30, 45) Retry until it works and move on Changed environment Re-evaluate, update the prompt, or pin the model version (Lesson 67) Roll back without understanding what changed Often, Two Fixes Are Right In the refund example:

Fix the code rule (<= 7). This removes the cause. Add a cross-step validation that a reply must not promise a refund when the decision is not eligible (Lesson 73). This adds a safety net for future errors of the same kind. The second fix alone would have hidden the bug; the first fix alone leaves the chain without a safety net.

Step 6: Add a Regression Test Every bug that reaches debugging should become a test case: the failing input, with the expected outputs of the steps involved. Run these tests whenever prompts, code, or models change, so the same bug cannot quietly return (Lesson 118).

REGRESSION_CASES = [ { "name": "refund requested after 10 days is not eligible", "state": {"days_since_purchase": 10}, "expected_decision": "refund_not_eligible", }, { "name": "refund requested on day 7 is eligible (boundary)", "state": {"days_since_purchase": 7}, "expected_decision": "refund_eligible", }, ]

for case in REGRESSION_CASES: result = decide_fixed(case["state"]) status = "PASS" if result == case["expected_decision"] else "FAIL" print(f"{status}: {case['name']}") Output:

PASS: refund requested after 10 days is not eligible PASS: refund requested on day 7 is eligible (boundary) The second case tests the boundary, exactly 7 days, which is where rule errors such as < versus <= usually appear.

Debugging Tools to Build Into Every Chain Tool Purpose Lesson Trace with inputs and outputs per step See the chain's work 62 Prompt and model version in every record Know exactly what ran 67 Validation results in the trace See which checks passed or failed 73 Step replay Test one step in isolation This lesson Expectations for test inputs Find the first failing step automatically This lesson Regression test set Prevent fixed bugs from returning 118 Note on privacy: Traces contain inputs and outputs, which may include personal data. Decide what to store, who can access traces, and how long they are kept. For sensitive data, store references or masked values instead of full text.

Common Debugging Patterns "The Final Answer Is Wrong, but Every Step Looks Fine" Check the handoffs. A step may be correct for the input it received, but the input may have been incomplete, truncated, or in the wrong format (Lesson 71).

"It Fails Only on Some Inputs" Group the failing inputs and look for what they share: language, length, a phrase pattern, a missing field. Shared characteristics usually point to a gap in a prompt's rules or examples.

"It Fails Only Sometimes on the Same Input" This is model variance on a borderline case. Look at the different outputs across runs to see which decision is unstable, then add a rule or example that decides it (Lesson 32).

"It Used to Work" Compare prompt versions, model versions, and code changes between the last good run and the first bad one (Lesson 67). Something in the environment changed.

Common Mistakes Mistake 1: Fixing the Step Where the Error Appears What it is: Changing the last step because that is where the wrong output became visible.

Why it happens: The final output is what users report.

What it causes: The symptom is hidden, and the real cause continues to affect other outputs.

How to fix it: Find the first step whose output is wrong.

Mistake 2: Debugging Without a Trace What it is: Trying to understand a failure from the final output alone.

Why it happens: Traces were not recorded.

What it causes: Guesswork and long debugging sessions.

How to fix it: Record inputs, outputs, versions, and validation results for every step.

Mistake 3: Ignoring Intermittent Failures What it is: Dismissing a failure because it does not reproduce on the next run.

Why it happens: The chain seems to work again.

What it causes: A real reliability problem reaches production.

How to fix it: Run the input many times and measure the failure rate.

Mistake 4: Changing Several Things at Once What it is: Editing two prompts and a validation rule in one attempt.

Why it happens: Wanting to fix the problem quickly.

What it causes: You cannot tell which change fixed the problem, or which one caused a new one.

How to fix it: Change one thing, replay, and test (Lesson 12).

Mistake 5: No Regression Test After the Fix What it is: Fixing the bug and moving on.

Why it happens: The fix seems obviously correct.

What it causes: A later change reintroduces the same bug unnoticed.

How to fix it: Turn every debugged failure into a test case, including its boundary values.

Best Practices Reproduce before fixing. Collect the input, versions, and trace, and measure how often it fails.

Read the trace from the beginning. The first wrong output is where the problem started.

Replay individual steps. Test hypotheses in isolation, without the rest of the chain.

Classify the root cause. Prompt, input, handoff, validation, code, variance, or environment.

Fix the cause, and add a safety net. Correct the source; strengthen validation for similar future errors.

Change one thing at a time. Keep cause and effect clear.

Add regression tests for every fixed bug. Include boundary cases.

Try It Yourself Exercise 1: Find the First Failing Step Write a trace for a three-step chain that summarizes a meeting and emails action items, where the summary step dropped one action item and the email step faithfully emailed the incomplete list. Write expectations and use first_failing_step() to locate the problem.

Exercise 2: Classify Root Causes For each scenario, identify the root cause category and the right fix:

The extraction step returns "days": "ten" and the decision step crashes. A classifier labels "Great course but please refund me" as Feedback in 4 of 10 runs. After switching to a newer model, replies have become much longer. The reply step promises delivery in 2 days, but no step ever provided a delivery time. Exercise 3: Replay a Model Step Take a model-based step from a chain you have built. Save the exact input of a case where it failed. Replay the step 10 times with that input and record how many outputs are wrong. Then make one prompt change, replay 10 more times, and compare the failure rates.