Prompt Chains for Summarization and Extraction

Why These Two Tasks Benefit From Chaining Summarization (Lesson 20) and information extraction (Lesson 22) are among the most common uses of language models. Single prompts work well for short, clean inputs. For long documents, many documents, or results that must be accurate and auditable, chains work better, because:

Long inputs must be split into chunks, and the results from each chunk must be combined. Faithfulness (no invented or changed facts) is easier to guarantee when facts are extracted and checked before a summary is written. Counts and statistics across many documents should be computed in code, not estimated by a model. Conflicting information across parts of a document must be detected, not silently resolved. This lesson presents four practical chain patterns for these tasks, with code.

Pattern Use It When

1. Extract, then summarize The summary must be faithful to specific facts
2. Chunked extraction with merging The document is too long for one prompt, and entities appear in several parts
3. Extract per item, aggregate in code, then summarize You need statistics over many documents
4. Summarize, then verify You need a fluent summary with a quality check Pattern 1: Extract, Then Summarize The Idea Instead of asking the model to summarize raw text directly, first extract the key facts into a structured form, check them, and then write the summary from the facts.

Document → [Extract facts as JSON] → [Validate facts] → [Write summary from facts] → Summary Why It Improves Faithfulness A direct summary is written while reading the whole document, and the model may merge, change, or add details (Lesson 20). When the summary is written from a short list of verified facts, there is much less room for drift, and every statement in the summary can be traced to a fact (Lesson 71, pass facts, not retellings).

Example Step 1 prompt (extract):

Extract the key facts from the announcement in as JSON with keys: course, start_date (YYYY-MM-DD), schedule, fee (integer), early_bird_fee (integer or null), early_bird_deadline (YYYY-MM-DD or null), requirements (array of strings). Use null for anything not stated. Return only the JSON object. Step 2 (code): validate types and formats (Lesson 73).

Step 3 prompt (summarize from facts):

Write a 3-sentence summary for students using only the facts in . Include every date and amount exactly as given. Do not add anything else. Implementation Requirements: pip install openai and an OPENAI_API_KEY environment variable.

import json

from openai import OpenAI

client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account

EXTRACT = """Extract the key facts from the announcement in as JSON with keys: course, start_date (YYYY-MM-DD), schedule, fee (integer), early_bird_fee (integer or null), early_bird_deadline (YYYY-MM-DD or null), requirements (array of strings). Use null for anything not stated. Return only the JSON object."""

SUMMARIZE = """Write a 3-sentence summary for students using only the facts in . Include every date and amount exactly as given. Do not add anything else."""

def call(instructions: str, text: str) -> str: response = client.responses.create(model=MODEL, instructions=instructions, input=text) return response.output_text.strip()

def extract_then_summarize(document: str) -> tuple[dict, str]: facts = json.loads(call(EXTRACT, f"\n{document}\n")) if not isinstance(facts.get("fee"), int): raise ValueError("Extraction failed: fee is missing or not an integer") summary = call(SUMMARIZE, f"\n{json.dumps(facts, ensure_ascii=False, indent=2)}\n") return facts, summary Code Explanation Step 1 produces structured facts with normalized dates and integer amounts (Lesson 22). A minimal check confirms the most important field before continuing. In a full chain, use complete validation (Lesson 73). Step 2 receives only the facts, formatted as readable JSON, not the original document. It cannot introduce details from parts of the document that were not extracted, and it cannot paraphrase the original wording into something different. The function returns both the facts and the summary, so the summary can be checked against the facts automatically, for example with the number check from Lesson 20. Trade-Off The summary contains only what the extraction schema captured. If the schema omits something important, the summary will too. Design the schema around what the summary must cover.

Pattern 2: Chunked Extraction With Merging The Problem A long document, such as a 40-page meeting transcript or a long contract, is split into chunks for processing (Lesson 15). Extraction is run on each chunk. But the same entity often appears in several chunks:

The same person is mentioned in chunks 1, 2, and 3, with different details in each. A name is written differently: "Priya Raman" and "priya raman". A detail changes: "Mentor" in chunk 1, "Lead Mentor" in chunk 3. Simply concatenating the results produces duplicates and hides conflicts.

The Chain Document → split into chunks ↓ [Extract per chunk] (parallel) ↓ [Merge in code: deduplicate, fill gaps, report conflicts] ↓ Merged records + conflict list Merging in Code def merge_records(chunk_results: list[list[dict]], key: str = "name") -> tuple[list[dict], list[str]]: """Merge records from several chunks by a key field, filling gaps and reporting conflicts.""" merged: dict[str, dict] = {} conflicts: list[str] = [] for chunk_index, records in enumerate(chunk_results, start=1): for record in records: k = record[key].strip().lower() if k not in merged: merged[k] = {**record, "sources": [chunk_index]} continue existing = merged[k] existing["sources"].append(chunk_index) for field, value in record.items(): if field == key or value is None: continue if existing.get(field) is None: existing[field] = value elif existing[field] != value: conflicts.append( f"{record[key]}: {field} is {existing[field]!r} in chunk " f"{existing['sources'][0]} but {value!r} in chunk {chunk_index}" ) return list(merged.values()), conflicts

chunks = [ [{"name": "Priya Raman", "role": "Mentor", "email": None}], [{"name": "priya raman", "role": "Mentor", "email": "priya@example.com"}, {"name": "Arun K", "role": "Student", "email": None}], [{"name": "Priya Raman", "role": "Lead Mentor", "email": "priya@example.com"}], ]

records, conflicts = merge_records(chunks) for record in records: print(record) for conflict in conflicts: print("Conflict:", conflict) Output:

{'name': 'Priya Raman', 'role': 'Mentor', 'email': 'priya@example.com', 'sources': [1, 2, 3]} {'name': 'Arun K', 'role': 'Student', 'email': None, 'sources': [2]} Conflict: Priya Raman: role is 'Mentor' in chunk 1 but 'Lead Mentor' in chunk 3 Code Explanation Matching key record[key].strip().lower() normalizes the name, so "Priya Raman" and "priya raman" are recognized as the same person. Real data may need stronger matching, such as removing titles or matching on email addresses.

sources Records which chunks mentioned each entity. This is provenance (Lesson 71): you can go back to those chunks to check details.

Filling gaps If the stored value is None and a later chunk provides a value, the gap is filled. Priya's email was missing in chunk 1 and found in chunk 2.

Reporting conflicts If two chunks give different non-null values, the conflict is recorded instead of silently choosing one. Priya's role is "Mentor" in chunk 1 but "Lead Mentor" in chunk 3. Perhaps she was promoted during the meeting, or one extraction was wrong. A person, or a dedicated follow-up prompt with both chunks, should decide.

Why Code Merges, Not the Model Merging by key, filling gaps, and detecting differences are exact operations. Code performs them reliably and explains every decision. A model asked to "combine these lists and remove duplicates" might merge different people, drop records, or resolve conflicts without telling you.

Pattern 3: Extract Per Item, Aggregate in Code, Then Summarize The Problem "Summarize what 300 students said in their course feedback" cannot be done well in one prompt: the input is too large, and a model asked to estimate proportions ("most students...") produces impressions, not numbers.

The Chain 300 feedback texts ↓ [Extract per item: topics and sentiment as JSON] (parallel, batched) ↓ [Aggregate in code: counts, percentages, representative examples] ↓ [Summarize the aggregates for a report] Why Each Step Is Placed Where It Is Step Executor Reason Per-item extraction Model Understanding each feedback text requires language understanding Counting and percentages Code Exact computation (Lesson 24) Choosing example quotes Code Select real quotes by topic, so examples are genuine Writing the report Model Turning numbers into readable prose Example Aggregation Output (Input to the Final Step) { "total_feedback": 300, "topics": { "Content": {"mentions": 142, "negative_share": 0.18}, "Instructor": {"mentions": 121, "negative_share": 0.09}, "Platform": {"mentions": 88, "negative_share": 0.61}, "Pricing": {"mentions": 40, "negative_share": 0.45} }, "example_quotes": { "Platform": ["Videos buffer every evening.", "The app logs me out daily."] } } The final prompt is asked to write a report from these numbers, with an instruction to state every percentage exactly as given. Because the numbers were computed in code, the report's statistics are correct by construction.

Pattern 4: Summarize, Then Verify The Idea When a fluent, natural summary of the original text is required, and Pattern 1's fact-based approach is too restrictive, write the summary directly and then add a verification step:

Document → [Summarize] → [Verify against the document] → OK? → publish ↓ no [Revise with the list of problems] → verify again Verification Methods Check How Lesson Numbers and dates appear in the source Code: extract numbers from the summary and look for them in the source 20 Each sentence is supported Model: label each summary sentence SUPPORTED or UNSUPPORTED 20 Required points are covered Code or model: check a list of required facts 57 Limiting the Loop A summarize-verify-revise loop must have a maximum number of revisions (for example, two). If verification still fails, the summary is sent for human review rather than revised indefinitely (Lesson 60).

Choosing a Pattern Situation Recommended Pattern Short announcement, summary must contain exact facts 1. Extract, then summarize Long transcript, need a list of people and their roles 2. Chunked extraction with merging Hundreds of reviews, need a report with statistics 3. Extract, aggregate in code, summarize Article summary for readers, must sound natural 4. Summarize, then verify Long document summary Map-reduce (Lesson 15), optionally combined with Pattern 4 Patterns can be combined. For example, a long-document summary can use map-reduce for the summary and Pattern 2 for extracting the people and decisions mentioned, with the extracted decisions passed to the final summary step.

Common Mistakes Mistake 1: Concatenating Chunk Results What it is: Joining extraction results from all chunks without merging.

Why it happens: Each chunk's output looks correct on its own.

What it causes: Duplicate entities and hidden conflicts.

How to fix it: Merge by a normalized key, fill gaps, and report conflicts.

Mistake 2: Asking the Model for Statistics What it is: Asking a prompt to estimate how many reviews mention something.

Why it happens: It is one prompt instead of a chain.

What it causes: Impressionistic numbers that look precise.

How to fix it: Extract per item, count in code, and give the counts to the summary step.

Mistake 3: Summarizing From Unchecked Extractions What it is: Passing extracted facts to the summary step without validation.

Why it happens: Extraction seems reliable.

What it causes: A wrong date or amount appears confidently in the summary.

How to fix it: Validate extracted facts before summarizing (Lesson 73).

Mistake 4: Silently Resolving Conflicts What it is: Choosing the first or last value when chunks disagree.

Why it happens: It produces a clean result.

What it causes: Wrong information with no indication that a conflict existed.

How to fix it: Report conflicts and resolve them deliberately.

Mistake 5: Endless Verification Loops What it is: Revising a summary until verification passes, with no limit.

Why it happens: The loop is written for the success case.

What it causes: High costs on inputs that cannot be summarized cleanly.

How to fix it: Limit revisions and escalate to human review.

Best Practices Extract facts before summarizing when accuracy matters. Summaries from verified facts cannot drift far.

Merge chunk results in code. Deduplicate, fill gaps, and report conflicts.

Keep provenance. Record which chunk or item each fact came from.

Compute statistics in code. Let the model write about numbers, not estimate them.

Verify fluent summaries. Check numbers in code and claims with a model.

Limit loops and escalate. Unresolved problems go to people.

Try It Yourself Exercise 1: Extract, Then Summarize Run extract_then_summarize() on three course announcements. Then summarize the same announcements directly with a single prompt. Compare both summaries against the original text and count any changed or missing dates and amounts.

Exercise 2: Merge Chunk Results Split a long meeting transcript (real or written by you) into four chunks. Extract the people mentioned, with role and email, from each chunk. Merge the results with merge_records() and review every conflict. Decide how you would resolve each one.

Exercise 3: Feedback Report Collect or write 30 short course feedback texts. Build Pattern 3: extract topics and sentiment per text as JSON, compute counts and negative shares with Python's Counter, and ask a model to write a 150-word report from the computed numbers. Check that every number in the report matches your computed values.