Retrieval-Augmented Generation (RAG)
Introduction What Is RAG? Retrieval-Augmented Generation (RAG) is a technique in which an application first retrieves relevant information from a collection of documents, then gives that information to a language model, which generates an answer based on it.
User question ↓ Retrieve: find the most relevant passages in your documents ↓ Augment: insert those passages into the prompt ↓ Generate: the model answers using the passages ↓ Answer (ideally with sources) In simple terms: instead of asking the model to answer from memory, you look up the relevant pages first and hand them to the model with the question, like an open-book exam.
The term comes from the 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by Lewis et al. Today, "RAG" is used broadly for any system that retrieves information and includes it in a model's prompt.
Why RAG Is Needed A language model's knowledge comes from its training data (Lesson 2). That creates three problems for real applications:
Problem Example How RAG Helps Private knowledge Your refund policy, course schedule, internal documentation The documents are retrieved and provided at question time Outdated knowledge A policy changed last month Update the documents; answers change immediately Unverifiable answers The model states a rule with no source Answers can cite the retrieved passages Lesson 19 showed open-book question answering, where you paste the relevant text into the prompt yourself. RAG automates the "find the relevant text" step, so it works for document collections far too large to paste.
RAG Compared With Alternatives Approach How It Adds Knowledge Best For Limitations Long prompt (all documents) Paste everything into the context A few short documents Cost, latency, and context limits grow with size; important details can be overlooked RAG Retrieve only relevant passages per question Large or changing document collections; answers needing sources Retrieval can miss the right passage Fine-tuning Train the model further on examples Teaching style, format, or specialized behavior Not ideal for frequently changing facts; costly to update RAG and fine-tuning solve different problems. For facts that change, such as policies, prices, and schedules, RAG is usually the better fit, because updating a document is easier than retraining a model.
The Components of a RAG System
Ingestion: Preparing the Documents Documents (web pages, PDFs, help articles, database records) are loaded and cleaned: removing navigation menus, headers, and other noise.
Chunking: Splitting Into Passages Documents are split into smaller pieces called chunks, typically a few sentences to a few paragraphs long. Retrieval works at the chunk level, so a question about refunds retrieves the refund paragraph, not the entire 20-page handbook.
Indexing: Making Chunks Searchable Chunks are stored in a way that allows fast search. Two common approaches:
Approach How It Finds Matches Strength Weakness Keyword search Shared words between question and chunk (for example TF-IDF or BM25) Exact terms, names, codes Misses synonyms ("money back" vs "refund") Semantic search Similar meaning, using embeddings Synonyms and paraphrases Can miss exact identifiers; needs an embedding model Many production systems combine both, which is called hybrid search.
Retrieval: Finding Relevant Chunks For each question, the system finds the top few chunks, often 3 to 10, that best match it.
Augmentation: Building the Prompt The retrieved chunks are inserted into a prompt, with instructions on how to use them (Lessons 103–106).
Generation: Answering The model writes the answer from the provided chunks, ideally citing which chunk each statement comes from.
Building a Minimal RAG Retriever The following example implements chunking and keyword-based retrieval with TF-IDF, so you can see how retrieval works without any API.
TF-IDF (term frequency–inverse document frequency) scores words by how often they appear in a chunk, while giving less weight to words that appear in many chunks. The similarity between a question and a chunk is then measured with cosine similarity, a score from 0 (nothing in common) to 1 (identical word profile).
Requirements: pip install scikit-learn
import re
from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity
def chunk_by_sentences(text: str, max_words: int = 30) -> list[str]: """Group whole sentences into chunks of at most about max_words words, repeating the last sentence of each chunk at the start of the next one.""" sentences = re.split(r"(?<=[.!?])\s+", text.strip()) chunks, current = [], [] for sentence in sentences: if current and len(" ".join(current + [sentence]).split()) > max_words: chunks.append(" ".join(current)) current = [current[-1]] # overlap: repeat the last sentence current.append(sentence) if current: chunks.append(" ".join(current)) return chunks
DOCUMENTS = { "refund-policy": "Refunds are available within 7 days of purchase. A refund is only possible " "if less than 20% of the course has been completed. Approved refunds reach " "the original payment method within 5 to 7 business days.", "certificates": "Certificates are issued after all modules and the final project are completed. " "Certificates can be downloaded from the dashboard as PDF files.", "live-classes": "Live classes run on Saturdays and Sundays from 7 PM to 9 PM IST. Recordings " "appear in the course library within two hours after each class.", "internship": "The free internship program is open to students who complete any advanced course. " "Interns work on real projects for eight weeks.", }
chunks = [ {"id": f"{doc_id}#{i}", "text": piece} for doc_id, text in DOCUMENTS.items() for i, piece in enumerate(chunk_by_sentences(text, max_words=30)) ]
vectorizer = TfidfVectorizer(stop_words="english") matrix = vectorizer.fit_transform([chunk["text"] for chunk in chunks])
def retrieve(question: str, k: int = 2, min_score: float = 0.05) -> list[tuple[float, str]]: scores = cosine_similarity(vectorizer.transform([question]), matrix)[0] ranked = sorted(zip(scores, chunks), key=lambda pair: pair[0], reverse=True)[:k] return [(round(float(score), 2), chunk["id"]) for score, chunk in ranked if score >= min_score]
for chunk in chunks: print(chunk["id"], "|", chunk["text"]) print() for question in ["How long does a refund take to reach my account?", "When are the live classes held?", "Can I get my money back?"]: print(question, "->", retrieve(question)) Output:
refund-policy#0 | Refunds are available within 7 days of purchase. A refund is only possible if less than 20% of the course has been completed. refund-policy#1 | A refund is only possible if less than 20% of the course has been completed. Approved refunds reach the original payment method within 5 to 7 business days. certificates#0 | Certificates are issued after all modules and the final project are completed. Certificates can be downloaded from the dashboard as PDF files. live-classes#0 | Live classes run on Saturdays and Sundays from 7 PM to 9 PM IST. Recordings appear in the course library within two hours after each class. internship#0 | The free internship program is open to students who complete any advanced course. Interns work on real projects for eight weeks.
How long does a refund take to reach my account? -> [(0.41, 'refund-policy#1'), (0.21, 'refund-policy#0')] When are the live classes held? -> [(0.36, 'live-classes#0')] Can I get my money back? -> [] Code Explanation chunk_by_sentences()
Splits text into sentences, then groups whole sentences until adding another would exceed max_words. Starts each new chunk with the last sentence of the previous chunk. This overlap keeps context that spans a chunk boundary: here, the refund rule about 20% appears in both refund chunks. Splitting on sentences avoids cutting a sentence in half, which would separate "Approved refunds" from "reach the original payment method". Chunk IDs IDs such as refund-policy#1 combine the document name and chunk number. They are used later for citations and debugging (Lesson 71, provenance).
TfidfVectorizer Builds a TF-IDF representation of every chunk. stop_words="english" ignores very common words such as "the" and "is", which carry little meaning.
retrieve()
Converts the question into the same representation and computes its cosine similarity with every chunk. Returns the top k chunks with scores at or above min_score, so unrelated chunks are not returned just to fill the list. What the Results Show "How long does a refund take..." correctly retrieves the chunk containing "5 to 7 business days" first. "When are the live classes held?" retrieves only the live-classes chunk; no other chunk passes the minimum score. "Can I get my money back?" retrieves nothing, even though the refund policy answers it. The question shares no meaningful words with the refund chunks: "money back" and "refund" mean the same thing but are different words. This last result is the key limitation of keyword retrieval, and it is why semantic search with embeddings exists.
Semantic Retrieval With Embeddings An embedding is a list of numbers, a vector, that represents the meaning of a text. Texts with similar meanings have similar vectors, even when they use different words. "Can I get my money back?" and "Refunds are available within 7 days" are close in embedding space.
Creating Embeddings With the OpenAI API Requirements: pip install openai numpy and an OPENAI_API_KEY environment variable.
import numpy as np from openai import OpenAI
client = OpenAI() EMBEDDING_MODEL = "text-embedding-3-small" # Check the models available in your account
def embed(texts: list[str]) -> np.ndarray: response = client.embeddings.create(model=EMBEDDING_MODEL, input=texts) return np.array([item.embedding for item in response.data])
chunk_vectors = embed([chunk["text"] for chunk in chunks])
def semantic_retrieve(question: str, k: int = 2) -> list[tuple[float, str]]: question_vector = embed([question])[0] scores = chunk_vectors @ question_vector / ( np.linalg.norm(chunk_vectors, axis=1) * np.linalg.norm(question_vector) ) top = np.argsort(scores)[::-1][:k] return [(round(float(scores[i]), 2), chunks[i]["id"]) for i in top]
print(semantic_retrieve("Can I get my money back?")) Code Explanation client.embeddings.create() returns one embedding per input text, in response.data, in the same order as the inputs. Chunk embeddings are computed once and stored. In real systems, they are saved in a vector database or a database with vector search support, so they are not recomputed for every question. Cosine similarity is computed with NumPy: the dot product (@) divided by the product of the vector lengths. np.argsort(scores)[::-1][:k] gives the indices of the highest scores. With embeddings, "Can I get my money back?" is expected to retrieve the refund chunks, because their meaning is similar even though the words differ. Exact scores depend on the embedding model.
From Retrieval to an Answer Retrieval is only half of RAG. The retrieved chunks must be used well in the prompt:
Answer the question in using only the sources in . Cite the source id in square brackets after each statement. If the sources do not contain the answer, say "I don't know based on the available documents."
Refunds are available within 7 days of purchase. ... ... Approved refunds reach the original payment method within 5 to 7 business days.
Can I get my money back? The following lessons cover this part in depth:
Lesson Topic 102 How prompt engineering improves each stage of RAG 103 Writing prompts that use retrieved context 104 Grounding answers in provided documents 105 Handling missing information 106 Reducing unsupported claims 138 Project: a RAG prompt for question answering over documents Where RAG Fails Understanding failure points helps you design and debug RAG systems.
Failure Point Example Typical Fix Ingestion PDF tables become scrambled text Better document parsing Chunking A rule and its exception end up in different chunks Sentence-aware chunking with overlap; larger chunks Retrieval The right chunk is not in the top results Embeddings, hybrid search, query rewriting (Lesson 102) Prompting The model ignores the sources or adds outside knowledge Clear grounding instructions (Lessons 103–104) Missing information The documents do not contain the answer Explicit fallback behavior (Lesson 105) Outdated documents The old policy is still indexed Remove or update old versions; include dates When a RAG answer is wrong, check in order: was the right chunk retrieved? If not, the problem is retrieval. If it was, the problem is in the prompt or the generation. This is the same "first failing step" approach used for debugging chains (Lesson 74).
Common Mistakes Mistake 1: Assuming RAG Makes Answers Correct What it is: Believing that adding retrieval eliminates errors.
Why it happens: Answers now come with sources.
What it causes: Errors from bad retrieval or ignored context go unnoticed.
How to fix it: Evaluate retrieval and answers separately, with test questions (Lessons 114–119).
Mistake 2: Chunks That Split Meaning What it is: Fixed-size chunks that cut sentences or separate related rules.
Why it happens: Splitting by character count is easy.
What it causes: Retrieved chunks lack the information needed to answer correctly.
How to fix it: Chunk by sentences or sections, with overlap.
Mistake 3: Keyword Search Only What it is: Relying on word matching for user questions.
Why it happens: It is simple to implement.
What it causes: Questions phrased differently from the documents retrieve nothing.
How to fix it: Use embeddings or hybrid search.
Mistake 4: No Minimum Relevance What it is: Always passing the top results to the model, even if they are unrelated.
Why it happens: The retriever always returns something.
What it causes: The model receives irrelevant context and may answer from it.
How to fix it: Use a minimum score, and handle "nothing relevant found" explicitly (Lesson 105).
Mistake 5: Indexing Outdated or Duplicate Documents What it is: Old policy versions remain in the index alongside new ones.
Why it happens: Documents are added but never removed.
What it causes: Conflicting or outdated answers.
How to fix it: Maintain the document collection, and include dates in chunk metadata (Lesson 103).
Best Practices Use RAG for private, changing, or citable knowledge. It keeps answers current and verifiable.
Chunk by meaning. Sentences or sections, with overlap.
Give every chunk a stable ID and metadata. For citations and debugging.
Combine keyword and semantic retrieval where possible. Each covers the other's weaknesses.
Filter weak matches. Irrelevant context harms answers.
Debug retrieval and generation separately. Find the first failing step.
Keep the document collection current. RAG answers are only as good as the documents.
Try It Yourself Exercise 1: Keyword Retrieval Add five documents from your own platform, such as FAQs about payments, internships, and certificates. Write ten questions, including five that use different words from the documents. Run retrieve() and record which questions retrieve the right chunk.
Exercise 2: Chunk Size Experiment Run the retrieval test with max_words set to 15, 30, and 80. For each setting, record how many questions retrieved a chunk containing the complete answer. Explain why very small and very large chunks both cause problems.
Exercise 3: Semantic Retrieval Use embed() and semantic_retrieve() on the same questions from Exercise 1. Compare which questions succeed with embeddings but failed with TF-IDF, and whether any question works better with keyword search, such as one containing an exact course code.