How Prompt Engineering Supports RAG

Prompts Appear Throughout a RAG System It is easy to think of RAG as "retrieval plus one answer prompt". In practice, prompts can improve almost every stage of a RAG pipeline, not just the final answer:

User question ↓ [Prompt] Rewrite into a standalone, searchable query ↓ [Prompt] Generate alternative queries or a hypothetical answer ↓ [Prompt] Extract filters (course, document type, date) ↓ Retrieval (keyword, semantic, or hybrid search) ↓ [Prompt] Rerank or filter retrieved chunks by relevance ↓ [Prompt] Compress chunks to the relevant sentences ↓ [Prompt] Generate a grounded answer with citations ↓ [Prompt] Check the answer against the sources Each prompt addresses a specific weakness of the basic pipeline from Lesson 101. This lesson explains each one, when it is worth adding, and how to keep it reliable. As always, add stages in response to observed failures, not all at once (Lesson 79).

Stage 1: Query Rewriting The Problem Users ask questions in conversation: "And what about the React one?" Such questions cannot be used for retrieval, because they depend on earlier messages.

The Prompt Rewrite the latest user message in as a standalone search query, using the conversation in to resolve references like "it", "that", or "the other one". Keep important terms such as course names, dates, and codes exactly. Return only the query. Why It Helps Retrieval The standalone query, such as "React Fundamentals course fee and early bird discount", contains the words and meaning that match the right documents. This prompt was introduced in Lesson 83 (Recipe 3); in RAG, it is often the single most valuable addition for chat-based assistants.

Stage 2: Multiple Queries The Problem A single phrasing may not match how the documents are written. Students write "money back"; the policy says "refund".

The Prompt Write 3 different search queries that would find documents answering the question in . Use different wording in each: one close to the user's words, one using formal or policy language, and one using related terms. Return only a JSON array of strings. Example Output ["get my money back course", "refund policy eligibility", "cancel enrollment payment return"] Combining the Results With Reciprocal Rank Fusion Each query produces its own ranked list of chunks. Reciprocal Rank Fusion (RRF), introduced by Cormack, Clarke, and Büttcher in 2009, combines several ranked lists into one. Each chunk receives a score of 1 / (k + rank) from every list it appears in, and the scores are added. Chunks that rank well in several lists rise to the top.

from collections import defaultdict

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]: """Combine several ranked lists of chunk IDs into one ranking.""" scores: dict[str, float] = defaultdict(float) for ranking in rankings: for rank, chunk_id in enumerate(ranking, start=1): scores[chunk_id] += 1 / (k + rank) return sorted(scores, key=lambda chunk_id: scores[chunk_id], reverse=True)

rankings = [ ["refund-policy#1", "refund-policy#0", "live-classes#0"], # results for query 1 ["refund-policy#0", "payments#2", "refund-policy#1"], # results for query 2 ["payments#2", "refund-policy#0", "certificates#0"], # results for query 3 ] print(reciprocal_rank_fusion(rankings)) Output:

['refund-policy#0', 'payments#2', 'refund-policy#1', 'live-classes#0', 'certificates#0'] Code Explanation 1 / (k + rank) gives a high score for a top rank and smaller scores for lower ranks. The constant k = 60 is the value used in the original paper; it reduces the dominance of the very first position. refund-policy#0 appears in all three lists (ranks 2, 1, and 2), so it receives the highest combined score. payments#2 appears in two lists (ranks 2 and 1) and narrowly beats refund-policy#1 (ranks 1 and 3). RRF only needs ranks, not scores, so it can combine results from different search methods, such as keyword and semantic search in hybrid retrieval. Cost Consideration Multiple queries mean multiple searches and an extra model call. Use this technique when evaluation shows that relevant chunks are often missed with a single query.

Stage 3: Hypothetical Document Embeddings (HyDE) The Idea A question and its answer often look different: "How long do refunds take?" versus "Approved refunds reach the original payment method within 5 to 7 business days." HyDE, introduced by Gao et al. (2022) in "Precise Zero-Shot Dense Retrieval without Relevance Labels", asks a model to write a hypothetical answer passage first, then uses that passage, rather than the question, for semantic search.

The Prompt Write a short passage, 2 or 3 sentences, that would appear in an online course platform's help documentation and answer the question in . Write it in the style of a policy or help article. Why It Works The hypothetical passage resembles the style and vocabulary of real documentation, so its embedding is closer to the embeddings of the relevant chunks.

An Important Caution The hypothetical passage may contain wrong facts, such as "refunds take 3 days". That is acceptable only because it is used for searching, never for answering. The answer must always come from the real retrieved documents. Never show the hypothetical passage to users or include it in the answer prompt.

Stage 4: Extracting Filters The Problem Many questions contain information that should narrow the search rather than be matched as text: "What is the refund policy for the Django course?" should search only documents about Django, or general policies, not every course page.

The Prompt Extract search filters from the question in .

Return only JSON with these keys:

• "course": one of {allowed_courses}, or null
• "doc_type": one of "policy", "faq", "course_page", "announcement", or null

Use null when the question does not clearly specify a value. Validating Filters in Code Filters directly control what the search can find. A wrong filter, such as a misspelled course name, can hide the right documents completely. Validate every value against allowed values, and drop anything unknown:

ALLOWED_COURSES = {"Python Basics", "Django REST APIs", "React Fundamentals"} ALLOWED_DOC_TYPES = {"policy", "faq", "course_page", "announcement"}

def validate_filters(filters: dict) -> dict: """Keep only filters with allowed values; drop everything else.""" clean = {} if filters.get("course") in ALLOWED_COURSES: clean["course"] = filters["course"] if filters.get("doc_type") in ALLOWED_DOC_TYPES: clean["doc_type"] = filters["doc_type"] return clean

print(validate_filters({"course": "Django REST APIs", "doc_type": "policy"})) print(validate_filters({"course": "Django", "doc_type": "blog", "author": "x"})) Output:

{'course': 'Django REST APIs', 'doc_type': 'policy'} {} Code Explanation Only filters with known values are kept. "Django" is not an exact course name, "blog" is not an allowed document type, and author is not a supported filter, so all are dropped. Dropping an invalid filter makes the search broader, which is safer than making it wrong. Structured outputs with enums (Lesson 99) reduce invalid values, but validation in code is still necessary because filters determine what users can see.

Stage 5: Routing The Problem Not every question needs retrieval, and not every question should search the same documents.

The Prompt Decide how to handle the question in :

• "docs": needs information from the platform's documents
• "account": needs the user's own data (orders, progress), which requires a tool call
• "general": a general programming question that does not need platform documents
• "unclear": the question needs clarification

Return only the label. Why It Helps Routing avoids unnecessary retrieval for general questions, sends account questions to secure tools that check the user's identity (Lesson 78), and asks for clarification instead of guessing (Lesson 83, Recipe 6).

Stage 6: Reranking The Problem Retrieval returns chunks that are similar to the question, but similarity is not the same as usefulness. A chunk about "refunds for workshops" may rank highly for a question about course refunds.

The Prompt Rate how useful each source in is for answering the question in : 3 = directly answers it, 2 = partly answers it, 1 = related but does not answer it, 0 = not related.

Return only JSON: [{"id": "...", "score": 0-3}] Using the Scores Keep only sources with scores of 2 or 3, ordered by score. This removes misleading chunks before the answer prompt. Dedicated reranking models, often called cross-encoders, are faster and cheaper than general language models for this task; a prompt-based reranker is a simple way to start and to measure whether reranking helps.

Stage 7: Context Compression The Problem Retrieved chunks often contain only one or two relevant sentences surrounded by unrelated text. Long, noisy context increases cost and can distract the model.

The Prompt From each source in , copy only the sentences that help answer the question in , word for word. Keep each source's id. Omit sources with no relevant sentences. Return only JSON: [{"id": "...", "sentences": ["..."]}] Verify the Copies Compressed sentences must be exact copies. Check that each sentence appears in its source chunk (Lesson 70), so the compression step cannot introduce invented text.

Stage 8: Generation and Stage 9: Checking The answer prompt, which tells the model how to use the retrieved context, is the subject of Lessons 103 to 106. A final checking prompt can label each statement in the answer as supported or unsupported by the sources (Lesson 20), which is covered in Lesson 106.

Which Stages to Add First Observed Problem Stage to Add Follow-up questions in chat fail 1. Query rewriting Relevant chunks are often missing from results 2. Multiple queries, or 3. HyDE Results include documents from the wrong course or type 4. Filters General questions trigger irrelevant retrieval 5. Routing Top results include related but unhelpful chunks 6. Reranking Prompts are long and answers include irrelevant details 7. Compression Answers contain claims not in the sources 9. Checking, plus Lessons 104–106 Measure the effect of each stage on a test set of questions (Lessons 115–119). Every stage adds latency and cost, so keep only the stages that improve results.

Common Mistakes Mistake 1: Only Improving the Answer Prompt What it is: Tuning the final prompt when the real problem is that the right chunks were never retrieved.

Why it happens: The answer prompt is the visible part of the system.

What it causes: Endless prompt changes that cannot fix missing information.

How to fix it: Check retrieval first: is the correct chunk in the results?

Mistake 2: Using Hypothetical Text as Facts What it is: Showing or using HyDE passages in answers.

Why it happens: They look like real documentation.

What it causes: Invented facts presented to users.

How to fix it: Use hypothetical passages only for search.

Mistake 3: Trusting Extracted Filters What it is: Passing model-extracted filters directly to search.

Why it happens: Filters are usually correct.

What it causes: Relevant documents are excluded by a wrong filter.

How to fix it: Validate filters against allowed values and drop unknown ones.

Mistake 4: Unverified Compression What it is: Accepting compressed context without checking that sentences are exact copies.

Why it happens: Compression outputs look like quotes.

What it causes: Paraphrased or invented sentences become "sources".

How to fix it: Verify each compressed sentence against its original chunk.

Mistake 5: Adding Every Stage at Once What it is: Building a complex pipeline before measuring the basic one.

Why it happens: Advanced techniques seem necessary.

What it causes: High latency and cost, with unclear benefits.

How to fix it: Start simple, measure failures, and add stages one at a time.

Best Practices Diagnose the stage that fails. Retrieval failures need retrieval fixes.

Rewrite conversational questions. Standalone queries retrieve better.

Use multiple queries or HyDE for vocabulary mismatches. Combine results with RRF.

Validate filters in code. Never let a wrong filter hide the right documents.

Rerank and compress with verification. Keep only useful, exactly copied content.

Keep generated search aids out of answers. Hypothetical passages are for retrieval only.

Measure every added stage. Keep what improves results; remove what does not.

Try It Yourself Exercise 1: Multiple Queries Using the retriever from Lesson 101, generate three queries for five questions that use different words from your documents. Retrieve results for each query, combine them with reciprocal_rank_fusion(), and compare with single-query retrieval.

Exercise 2: Filter Extraction Write ten questions, some mentioning a specific course and document type, some not. Extract filters with the Stage 4 prompt, run validate_filters(), and check that every dropped filter was indeed invalid.

Exercise 3: Reranking Retrieve the top five chunks for five questions, then rerank them with the Stage 6 prompt. Record how often the most useful chunk moves to the top, and whether any useful chunk was wrongly scored below 2.