Prompt Chaining Fundamentals
The Smallest Useful Chain Lesson 61 introduced prompt chaining with a multi-step support workflow, and Lessons 62–63 covered how to design and decompose larger chains. This lesson returns to the fundamentals with the smallest useful chain: two prompts, where the second uses the first one's output. Understanding this pattern thoroughly makes every larger chain easier to build.
Input → [Prompt 1] → intermediate output → [Prompt 2] → final output Three ideas are at the core of every chain:
Idea Meaning Handoff The intermediate output is inserted into the next prompt Gate Code checks the intermediate output before passing it on, and can stop the chain Focus Each prompt has a narrower job than the whole task A Classic Two-Step Chain: Answering Questions From a Document A well-known chaining pattern for document question answering splits the task into two prompts:
Extract the passages from the document that are relevant to the question. Answer the question using only those passages. Why Not Use One Prompt? A single prompt, "Answer the question using the document", works for short documents (Lesson 19). For long documents, splitting the task helps:
Step 1 has one narrow job: find relevant text. It does not need to compose an answer. Step 2 sees only relevant text, so unrelated sections cannot distract it. The intermediate output is checkable: you can verify that the extracted quotes actually exist in the document before any answer is written. If nothing relevant is found, the chain can stop and return "not available" without asking the model to answer at all. Step 1: The Extraction Prompt Find the sentences in the document in that are relevant to the question in .
Copy each relevant sentence exactly, word for word, inside its own tags, all inside tags. If no sentence is relevant, return NONE. Do not answer the question. Design Decisions Decision Reason "Copy exactly, word for word" Exact quotes can be verified against the document in code One per sentence Each quote can be parsed and checked separately NONE when nothing is relevant Gives the chain a clear signal to stop "Do not answer the question" Keeps step 1 focused; answering is step 2's job Example Document:
Refunds are available within 7 days of purchase. A refund is only possible if less than 20% of the course has been completed. Refunds are processed to the original payment method within 5 to 7 business days. Certificates are issued after all modules and the final project are completed. Question: Can I get a refund if I bought the course 10 days ago?
Possible step 1 output:
Refunds are available within 7 days of purchase. The certificate sentence and the processing-time sentence are correctly left out: they do not help answer this question.
The Gate: Checking the Handoff in Code Before step 2 runs, code checks the intermediate output. This is the gate.
Parsing and Verifying Quotes import re
def parse_quotes(text: str) -> list[str]: """Return the text of every ... in the step 1 output.""" return [q.strip() for q in re.findall(r"(.*?)", text, re.DOTALL)]
def normalize(text: str) -> str: """Collapse all whitespace so line breaks do not prevent matching.""" return " ".join(text.split())
def verified_quotes(quotes: list[str], document: str) -> list[str]: """Keep only quotes that actually appear in the document.""" normalized_document = normalize(document) return [q for q in quotes if normalize(q) in normalized_document]
document = """Refunds are available within 7 days of purchase. A refund is only possible if less than 20% of the course has been completed. Refunds are processed to the original payment method within 5 to 7 business days."""
step1_output = """ Refunds are available within 7 days of purchase. A refund is only possible if less than 20% of the course has been completed. Refunds are always approved for students. """
quotes = parse_quotes(step1_output) print(len(quotes), verified_quotes(quotes, document)) print(parse_quotes("NONE")) Output:
3 ['Refunds are available within 7 days of purchase.', 'A refund is only possible if less than 20%\nof the course has been completed.'] [] Code Explanation parse_quotes() Finds the text between each pair of tags. The non-greedy (.*?) stops at the first closing tag, and re.DOTALL allows quotes that span several lines. When the output is NONE, there are no tags, so the result is an empty list.
normalize() Replaces every run of whitespace, including line breaks, with a single space. The second quote in the example was split across two lines by the model; without normalization, it would not match the document.
verified_quotes() Keeps only quotes that appear in the normalized document. The third quote, "Refunds are always approved for students.", does not exist in the document: it is an invented quote, and the gate removes it before it can influence the answer.
What the Gate Decides Gate Result Chain Action At least one verified quote Continue to step 2 with the verified quotes only No quotes, or none verified Stop and return "The information is not available in the document." Some quotes removed as unverified Continue, and log the removed quotes for review The gate turns an unchecked handoff into a verified one. This is the main practical advantage of chaining over a single prompt.
Step 2: The Answer Prompt Answer the question in using only the quotes in . If the quotes do not fully answer the question, say what is missing. Keep the answer under 60 words. Why Step 2 Receives Quotes, Not the Document Step 2 only sees text that was verified as relevant and real. It cannot use other parts of the document, and it cannot be influenced by invented quotes, because those were removed by the gate.
Possible Step 2 Output No. Refunds are available only within 7 days of purchase, and your purchase was 10 days ago. The Complete Chain in Python Requirements: pip install openai and an OPENAI_API_KEY environment variable. The code uses parse_quotes() and verified_quotes() from above.
from openai import OpenAI
client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account
EXTRACT = """Find the sentences in the document in that are relevant to the question in . Copy each relevant sentence exactly, word for word, inside its own tags, all inside tags. If no sentence is relevant, return NONE. Do not answer the question."""
ANSWER = """Answer the question in using only the quotes in . If the quotes do not fully answer the question, say what is missing. Keep the answer under 60 words."""
NOT_AVAILABLE = "The information is not available in the document."
def call(instructions: str, text: str) -> str: response = client.responses.create(model=MODEL, instructions=instructions, input=text) return response.output_text.strip()
def answer_from_document(document: str, question: str) -> dict: question_block = f"\n{question}\n"
# Step 1: extract step1 = call(EXTRACT, f"\n{document}\n\n\n{question_block}")
# Gate: verify quotes = parse_quotes(step1) kept = verified_quotes(quotes, document) removed = [q for q in quotes if q not in kept] if not kept: return {"answer": NOT_AVAILABLE, "quotes": [], "removed": removed}
# Step 2: answer quotes_block = "\n" + "\n".join(f"{q}" for q in kept) + "\n" answer = call(ANSWER, f"{quotes_block}\n\n{question_block}") return {"answer": answer, "quotes": kept, "removed": removed}
policy = """Refunds are available within 7 days of purchase. A refund is only possible if less than 20% of the course has been completed. Refunds are processed to the original payment method within 5 to 7 business days. Certificates are issued after all modules and the final project are completed."""
for question in [ "Can I get a refund if I bought the course 10 days ago?", "Can I transfer my course to a friend?", ]: print(answer_from_document(policy, question)) Possible output:
{'answer': 'No. Refunds are available only within 7 days of purchase, and your purchase was 10 days ago.', 'quotes': ['Refunds are available within 7 days of purchase.'], 'removed': []} {'answer': 'The information is not available in the document.', 'quotes': [], 'removed': []} Code Explanation Two prompts, one job each EXTRACT finds evidence; ANSWER writes the answer from evidence. Neither prompt has to do both.
The handoff Verified quotes are rebuilt into a block for step 2. Only verified text crosses from step 1 to step 2.
The early exit For the transfer question, step 1 finds nothing relevant, the gate stops the chain, and step 2 is never called. This saves a model call and guarantees that no answer is invented when the document is silent.
The return value Along with the answer, the function returns the quotes used and any quotes removed by the gate. The quotes can be shown to users as evidence, and the removed quotes are useful for monitoring how often step 1 invents text.
Chaining Without Code Chaining is not only for applications. In a chat interface, you can perform the same two steps manually:
Message 1:
Here is our refund policy: [policy text] List the exact sentences that are relevant to this question, without answering it: "Can I get a refund if I bought the course 10 days ago?" Message 2 (after checking the quotes yourself):
Using only those sentences, answer the question in under 60 words. You act as the gate: you read the quotes, confirm they are correct, and only then ask for the answer. This is a useful habit for any important question about a long document.
Fundamental Chain Shapes The two-step document chain is one example of a sequential chain. Every chain is built from a few basic shapes, which Lesson 61 introduced and later lessons combine:
Shape Structure Two-Step Example Sequential A → B Extract quotes → answer Gated A → check → B or stop Extract quotes → verify → answer or "not available" Branching A → B1 or B2 Classify message → billing reply or technical reply Parallel A1 and A2 → combine Summarize two documents → compare summaries Loop A → check → A again Draft → check → revise Most real chains combine these shapes. Recognizing them makes complex workflows easier to read and to design.
Common Mistakes Mistake 1: Passing Unverified Output Forward What it is: Inserting step 1's output into step 2 without checking it.
Why it happens: The handoff seems straightforward.
What it causes: Invented content from step 1 becomes "evidence" in step 2.
How to fix it: Add a gate that validates or verifies the intermediate output.
Mistake 2: Letting Step 1 Answer the Question What it is: The extraction prompt also writes an answer or a conclusion.
Why it happens: The instruction does not forbid it.
What it causes: Step 2 copies step 1's conclusion instead of reasoning from the evidence.
How to fix it: State clearly what step 1 must not do.
Mistake 3: No Stop Condition What it is: Always running step 2, even when step 1 found nothing.
Why it happens: The chain is designed for the successful path only.
What it causes: Step 2 receives empty or irrelevant input and may produce an invented answer.
How to fix it: Define an explicit signal, such as NONE, and stop the chain when it appears.
Mistake 4: Exact Matching Without Normalization What it is: Comparing quotes with the document character by character.
Why it happens: Exact matching seems strict and safe.
What it causes: Correct quotes are rejected because of line breaks or extra spaces.
How to fix it: Normalize whitespace on both sides before comparing.
Best Practices Start with two steps. Add more only when a step has a distinct job.
Make the intermediate output checkable. Exact quotes, tags, or JSON.
Put a gate between steps. Verify, filter, or stop.
Give each prompt a single job and say what it must not do. Extraction does not answer; answering does not search.
Pass only verified, relevant content forward. The next step should never see invalid input.
Return evidence with results. Quotes make answers transparent and debuggable.
Try It Yourself Exercise 1: Run the Chain Run answer_from_document() with a document of your own, such as an internship policy or a course FAQ, and ten questions: six answerable, two partly answerable, and two unanswerable. Check that unanswerable questions stop at the gate.
Exercise 2: Catch Invented Quotes Modify the extraction prompt to remove "exactly, word for word" and run the chain on the same questions. Count how many quotes the gate removes compared with the original prompt. Explain what this shows about writing checkable intermediate outputs.
Exercise 3: Compare With a Single Prompt For the same document and questions, use a single prompt: "Answer using only the document; if the answer is not in the document, say so." Compare its answers with the chain's answers. Note which approach handles unanswerable questions better, and how many model calls each uses.