Validating Intermediate Results in a Prompt Chain
Why Validate Between Steps? In a chain, every step trusts the output of the step before it. If one step produces a wrong value, every later step builds on it, and the final result can look polished while being wrong. Lesson 61 introduced validation between steps, and Lesson 70 showed a simple gate. This lesson treats validation as a design topic of its own: what to check, how to check it, and what to do when a check fails.
Step 1 → [Validate] → Step 2 → [Validate] → Step 3 → [Validate] → Result ↓ ↓ ↓ retry / fallback / retry / fallback / retry / fallback / escalate escalate escalate Where Errors Are Cheapest to Catch Where the Error Is Caught Cost Right after the step that caused it One retry of one step At the end of the chain Several wasted steps, and the cause is harder to find By a user after delivery Lost trust, support effort, possible harm Validation moves error detection as early as possible.
The Layers of Validation Different kinds of errors need different checks. A robust chain combines several layers, from cheap and exact to expensive and approximate.
Layer Question Example Check Done By
- Structure Is the output in the expected shape? Valid JSON; required fields present; correct types Code
- Values Is each value acceptable on its own? Fee within a plausible range; date in YYYY-MM-DD; label in the allowed set Code
- Consistency Do the values make sense together? Early-bird fee lower than the regular fee; end date after start date Code
- Grounding Do the values come from the source? Each extracted number appears in the input text Code
- Cross-step agreement Does this output agree with earlier steps? The reply mentions the same decision that the routing step made Code or model
- Meaning Is the output correct and appropriate? Every summary sentence is supported by the source Model or person Why Layers Matter Each layer catches errors the others miss:
Structure catches broken formats, but not wrong values. Values catches impossible numbers, but not plausible wrong ones. Consistency catches values that are each plausible but contradictory together, such as swapped fees. Grounding catches invented values, but not values copied from the wrong place. Meaning catches subtle errors, but is slower, costs more, and is itself imperfect. Apply the cheap code layers to every output, and add model-based meaning checks where the risk justifies the cost.
Implementing Layered Validation in Python The following validator checks an extraction step's output using layers 1 to 4. Each check returns a list of issues with a severity:
error: the output must not continue; retry, fall back, or escalate. warning: the output may continue, but the issue is logged for review. import re from dataclasses import dataclass
@dataclass class Issue: severity: str # "error" stops or retries the step; "warning" is logged message: str
SCHEMA = {"course": str, "fee": int, "early_bird_fee": int, "start_date": str}
def check_structure(data: dict) -> list[Issue]: issues = [] for field, kind in SCHEMA.items(): if field not in data: issues.append(Issue("error", f"Missing field: {field}")) elif data[field] is not None and not isinstance(data[field], kind): issues.append(Issue("error", f"{field} should be {kind.name}")) return issues
def check_values(data: dict) -> list[Issue]: issues = [] fee = data.get("fee") if isinstance(fee, int) and not 500 <= fee <= 100_000: issues.append(Issue("error", f"fee {fee} is outside the plausible range 500-100000")) date = data.get("start_date") if isinstance(date, str) and not re.fullmatch(r"\d{4}-\d{2}-\d{2}", date): issues.append(Issue("error", f"start_date {date!r} is not in YYYY-MM-DD format")) return issues
def check_consistency(data: dict) -> list[Issue]: issues = [] fee, early = data.get("fee"), data.get("early_bird_fee") if isinstance(fee, int) and isinstance(early, int) and early >= fee: issues.append(Issue("error", f"early_bird_fee {early} should be lower than fee {fee}")) return issues
def check_grounding(data: dict, source: str) -> list[Issue]: issues = [] numbers_in_source = re.findall(r"\d+", source.replace(",", "")) for field in ("fee", "early_bird_fee"): value = data.get(field) if isinstance(value, int) and str(value) not in numbers_in_source: issues.append(Issue("error", f"{field} {value} does not appear in the source text")) course = data.get("course") if isinstance(course, str) and course.lower() not in source.lower(): issues.append(Issue("warning", f"course name {course!r} does not appear verbatim in the source")) return issues
def validate(data: dict, source: str) -> list[Issue]: return (check_structure(data) + check_values(data)
- check_consistency(data) + check_grounding(data, source)) Testing the Validator source = "New batch: Django REST APIs, starts 15 Nov 2026, fee Rs. 4,999 (early bird 3,999)."
good = {"course": "Django REST APIs", "fee": 4999, "early_bird_fee": 3999, "start_date": "2026-11-15"} bad = {"course": "Django REST Framework", "fee": 3999, "early_bird_fee": 4999, "start_date": "15 Nov 2026"}
for data in (good, bad): print([f"{issue.severity}: {issue.message}" for issue in validate(data, source)]) Output:
[] ["error: start_date '15 Nov 2026' is not in YYYY-MM-DD format", 'error: early_bird_fee 4999 should be lower than fee 3999', "warning: course name 'Django REST Framework' does not appear verbatim in the source"] Code Explanation Issue A small dataclass with a severity and a message. Collecting issues, instead of raising an exception at the first problem, gives a complete picture of everything wrong with an output.
check_structure() Confirms that every field in SCHEMA exists and has the right type. None is allowed, because missing information should be null rather than invented (Lesson 22).
check_values() Checks single values: a plausible fee range, and the required date format using re.fullmatch, which requires the whole string to match.
check_consistency() Checks values together. In the bad example, the fees were swapped: both 3999 and 4999 appear in the source, so grounding alone would not catch this mistake. The consistency rule, an early-bird fee must be lower than the regular fee, does.
check_grounding()
Removes commas from the source, so "4,999" becomes "4999", and collects every number. Reports an error for any extracted fee that does not appear in the source, which catches invented values. Reports a warning for a course name that does not appear verbatim. "Django REST Framework" is not in the source; it may be a harmless rewording or an error, so it is logged rather than blocking the chain. Why grounding uses "appears in the source" This check confirms that a value exists in the source, not that it is attached to the right field. That is why consistency checks and, for important fields, meaning checks are still needed.
What to Do When Validation Fails Option 1: Retry With Feedback Send the step's prompt again, together with a description of what was wrong. Models often fix specific, clearly described errors on the second attempt.
def feedback_message(issues: list[Issue]) -> str: errors = [f"- {issue.message}" for issue in issues if issue.severity == "error"] return ("Your previous output had these problems:\n" + "\n".join(errors)
- "\nReturn a corrected JSON object.")
print(feedback_message(validate(bad, source))) Output:
Your previous output had these problems:
- start_date '15 Nov 2026' is not in YYYY-MM-DD format
- early_bird_fee 4999 should be lower than fee 3999 Return a corrected JSON object. Explanation: Only errors are included; warnings would distract from what must be fixed. The message names each problem precisely, which is far more useful than "Your output was invalid." This is the same principle as Reflexion (Lesson 60): specific feedback produces specific corrections.
Option 2: Fall Back Use a safer alternative: a different model, a simpler rule-based method, or a default value, if a default is safe in your context.
Option 3: Escalate Stop the chain and send the case to a person, with the output and the list of issues attached. This is the right choice when errors persist after a retry, or when the risk of continuing is high.
Choosing a Response Situation Response First failure, clear format or value error Retry with feedback Failure repeats after one or two retries Escalate Optional step fails Skip it, and log the failure Grounding error on a critical value (price, date, legal term) Escalate; do not guess Only warnings Continue and log Always limit retries per step, as in the chain runner from Lesson 62.
Cross-Step Validation Some errors only appear when comparing outputs from different steps.
Earlier Step Output Later Step Output Cross-Step Check Decision: refund_not_eligible Reply text The reply must not promise a refund Extracted fee: 4999 Course description The description must state ₹4,999, not another amount Selected label: Technical Routed queue The ticket must be routed to the technical team Extracted IDs: R1–R20 Classification results Every ID must have exactly one result Implementing Cross-Step Checks Many cross-step checks are simple code:
def reply_matches_decision(decision: str, reply: str) -> list[Issue]: promises_refund = re.search(r"\b(refund (is|has been) approved|we will refund)\b", reply, re.IGNORECASE) if decision == "refund_not_eligible" and promises_refund: return [Issue("error", "Reply promises a refund, but the decision was not eligible")] return [] Keyword checks like this catch clear contradictions but miss rephrased ones, such as "your money will be returned". For important decisions, combine code checks with a model-based check (Lesson 61, the CHECK prompt), or require human approval.
Model-Based Meaning Checks When correctness cannot be checked with code, a separate prompt can review an output against its source or requirements (Lesson 10, Tip 6; Lesson 20).
Compare the summary in
with the facts in . For each sentence in the summary, reply SUPPORTED or UNSUPPORTED, with a short reason. Using Meaning Checks Wisely They cost an extra model call and add latency, so apply them where errors matter most. They are imperfect: a checker can miss errors or flag correct statements. Validate the checker itself on examples with known answers (Lesson 69). Run code checks first. There is no point paying for a meaning check on output that is not even valid JSON. Logging Validation Results Record every validation result in the chain's trace (Lesson 62):Logged Use Step name, attempt number Which step and how many retries Issues with severity What went wrong Action taken Retried, fell back, escalated, continued Prompt version Link failures to a specific prompt (Lesson 67) Over time, these logs show which checks fail most often. A check that fails frequently points to a prompt that needs improvement, and a check that never fails may be unnecessary or too weak.
Common Mistakes Mistake 1: Validating Only the Format What it is: Checking that the output is valid JSON and nothing more.
Why it happens: Format checks are easy.
What it causes: Plausible but wrong values pass through the chain.
How to fix it: Add value, consistency, and grounding checks.
Mistake 2: Validating Only at the End What it is: Checking the final result but not intermediate outputs.
Why it happens: The final result is what users see.
What it causes: Errors are found late, cost more to fix, and are hard to trace.
How to fix it: Validate after every step that produces data used later.
Mistake 3: Vague Retry Feedback What it is: Retrying with "Your output was invalid, try again."
Why it happens: The validator returns only pass or fail.
What it causes: The same mistake is repeated.
How to fix it: Collect specific issue messages and include them in the retry.
Mistake 4: Treating Every Issue as Fatal What it is: Stopping the chain for minor issues such as a reworded course name.
Why it happens: All issues are handled the same way.
What it causes: Too many escalations, and people stop paying attention to them.
How to fix it: Use severities: errors stop or retry; warnings are logged.
Mistake 5: Trusting Model Checkers Blindly What it is: Assuming a model-based checker is always right.
Why it happens: It is "just a check".
What it causes: False confidence in outputs the checker failed to catch.
How to fix it: Measure the checker against known cases, and use code checks wherever possible.
Best Practices Validate after every step whose output is used later. Catch errors where they start.
Layer your checks. Structure, values, consistency, grounding, cross-step agreement, and meaning.
Run cheap code checks first. Use model checks only where they add value.
Use severities. Distinguish errors that block from warnings that are logged.
Retry with specific feedback, and limit retries. Then escalate.
Check outputs against earlier steps. Decisions, values, and IDs must stay consistent.
Log every validation result. Use the data to improve prompts and checks.
Try It Yourself Exercise 1: Extend the Validator Add two checks to the validator: a check_values rule that start_date must not be in the past (use datetime.date.today()), and a check_consistency rule that requires early_bird_fee to be null if the source does not contain the words "early bird". Test both with suitable inputs.
Exercise 2: Retry With Feedback Build a chain step that extracts course details with a model, validates the result with validate(), and retries once with feedback_message() if there are errors. Test it on five announcements written in different styles, and record how many needed a retry and whether the retry fixed the problem.
Exercise 3: Cross-Step Check Implement reply_matches_decision() in the ticket chain from Lesson 61. Write five replies by hand, including two that contradict the decision in different words ("we'll process your refund", "your money will be returned"). Check which ones the keyword rule catches, and design an additional check for the ones it misses.