Chunking Strategies for Better Retrieval

Chunking — splitting a long document into smaller pieces before embedding each piece separately — was mentioned briefly in Lesson 3 as the correct alternative to truncating long text. This lesson treats chunking as its own subject, because the specific strategy used has a direct, often large, effect on retrieval quality. Two pipelines that use the identical embedding model and the identical similarity search code can produce very different search results purely because of how they split documents into chunks.

Why Chunking Strategy Matters This Much

A retrieval system can only return what it has as a discrete, embedded unit. If an entire 20-page document is embedded as one vector, that vector represents an average of everything in the document — a query about one specific paragraph on page 14 has to compete, in that single vector, with the meaning of the other nineteen and a half pages. The result: a highly relevant document can score lower than it should, because its single embedding is diluted by unrelated content elsewhere in the same document.

Chunking fixes this by embedding smaller, more focused pieces — but it introduces a new problem: how the split is made determines whether each chunk is a coherent, self-contained unit of meaning, or an arbitrary fragment that cuts a sentence, an idea, or a code example in half. A badly chunked document can retrieve worse than one not chunked at all, because half-sentences and orphaned fragments embed poorly and read poorly even when they are retrieved correctly.

Strategy 1: Fixed-Size Chunking

The simplest approach: split text into chunks of a fixed length, measured in characters or tokens.

def fixed_size_chunks(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    """Split text into fixed-size character chunks with overlap.

    `overlap` characters from the end of each chunk are repeated at the
    start of the next chunk.
    """
    if overlap >= chunk_size:
        raise ValueError("overlap must be smaller than chunk_size")

    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

Why include overlap at all? Without overlap, a sentence or idea that happens to fall exactly on a chunk boundary is split between two chunks, and neither chunk alone contains the complete thought — a query about that specific idea may fail to match either half well. Overlap (here, overlap characters repeated at the start of the next chunk) means the region around a boundary appears intact in at least one of the two adjacent chunks, reducing the chance that a boundary destroys a specific piece of meaning. A typical overlap is 10-20% of the chunk size — enough to cover most sentence-boundary cases without doubling storage and embedding cost.

Why is fixed-size chunking's simplicity also its weakness? It splits purely by character count, with no awareness of sentence, paragraph, or section boundaries. A 500-character cutoff can and will land in the middle of a sentence, a code example, or a table row, producing chunks that are syntactically broken even if surrounding overlap helps semantically. It is fast, dependency-free, and works acceptably on unstructured, dense prose, but is rarely the best choice when better options are easy to implement.

Strategy 2: Sentence- and Paragraph-Aware Chunking

A better default respects natural text boundaries, only falling back to a hard cut when a single unit (a sentence, a paragraph) is itself too long.

import re


def split_into_sentences(text: str) -> list[str]:
    """A simple sentence splitter based on punctuation.

    Good enough for well-formed prose; a proper NLP sentence tokenizer
    (e.g. from spaCy or NLTK) handles more edge cases (abbreviations,
    decimal numbers) more reliably for production use on messy text.
    """
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())
    return [s for s in sentences if s]


def sentence_aware_chunks(text: str, max_chunk_chars: int = 500) -> list[str]:
    """Group whole sentences into chunks up to max_chunk_chars long."""
    sentences = split_into_sentences(text)
    chunks = []
    current = ""

    for sentence in sentences:
        candidate = f"{current} {sentence}".strip()
        if len(candidate) <= max_chunk_chars:
            current = candidate
        else:
            if current:
                chunks.append(current)
            current = sentence  # a single sentence longer than the budget stands alone

    if current:
        chunks.append(current)
    return chunks

sentence_aware_chunks accumulates whole sentences into current until adding the next sentence would exceed max_chunk_chars, at which point it closes off the current chunk and starts a new one — this guarantees every chunk boundary falls between sentences, never in the middle of one. The one exception: a single sentence longer than max_chunk_chars is kept intact rather than cut mid-sentence, accepting an oversized chunk in the rare case rather than breaking a sentence, which would defeat the purpose of this approach entirely.

Why is this generally better than fixed-size chunking for prose? Every chunk is a coherent set of complete sentences, which both embeds more meaningfully (the model receives intact ideas, not fragments) and reads better when shown directly to a user or included in a generation prompt. The trade-off is implementation complexity and a dependency on reasonably reliable sentence boundaries — text with unusual formatting (bullet lists, code blocks, tables) does not split cleanly by sentence punctuation, which motivates the next strategy.

Strategy 3: Structure-Aware (Recursive) Chunking

Real documents have structure beyond plain sentences: headings, paragraphs, list items, code blocks. Structure-aware chunking splits along the largest natural boundary first (sections, then paragraphs, then sentences), only descending to a smaller unit when a larger one is still too big.

def recursive_chunks(text: str, max_chunk_chars: int = 500,
                      separators: list[str] | None = None) -> list[str]:
    """Split text by progressively finer separators, only descending
    to a finer separator when a piece is still too large.
    """
    separators = separators or ["\n\n", "\n", ". ", " "]

    def split_piece(piece: str, remaining_separators: list[str]) -> list[str]:
        if len(piece) <= max_chunk_chars:
            return [piece] if piece.strip() else []
        if not remaining_separators:
            # No separator left small enough to help; hard-cut as a last resort.
            return [piece[i:i + max_chunk_chars] for i in range(0, len(piece), max_chunk_chars)]

        sep, rest = remaining_separators[0], remaining_separators[1:]
        parts = piece.split(sep)
        result = []
        buffer = ""
        for part in parts:
            candidate = (buffer + sep + part) if buffer else part
            if len(candidate) <= max_chunk_chars:
                buffer = candidate
            else:
                if buffer:
                    result.extend(split_piece(buffer, rest))
                buffer = part
        if buffer:
            result.extend(split_piece(buffer, rest))
        return result

    return split_piece(text.strip(), separators)

How this works, step by step: the function tries the coarsest separator first ("\n\n", a paragraph break). It groups consecutive paragraphs into a buffer as long as the buffer stays within max_chunk_chars. When a buffer would grow too large, it is finalized — but only after being recursively re-split using the next separator in the list ("\n", then ". ", then " ") if it is still too big on its own. This means a single oversized paragraph gets broken down into lines, and a single oversized line gets broken into sentences, and so on — the split only gets finer where it actually needs to, preserving larger natural boundaries (like whole paragraphs) wherever they already fit the size budget. The final fallback — a hard character cut — only triggers if a piece has no more separators left to try and is still too large (an unbroken run of text longer than the entire budget), which is rare in normal prose.

This is conceptually the same idea used by widely adopted recursive text splitters in RAG frameworks, implemented here from first principles so the mechanism is fully visible rather than hidden behind a library call.

Choosing Chunk Size: The Core Trade-off

Why is there no single correct chunk size? The right size depends on the content and the query pattern. Short, focused FAQ entries or single-fact snippets retrieve well as small chunks because each one already represents one complete idea. Long-form technical documentation, where understanding one paragraph often depends on the paragraph before it, benefits from somewhat larger chunks (or from including a small amount of surrounding context alongside each chunk, sometimes called "parent document" retrieval) so a returned chunk is not missing information needed to make sense of it. This is precisely why Lesson 9's evaluation methodology matters — chunk size and strategy should be selected by measuring retrieval quality on real, representative queries, not by assumption.

Preserving Chunk-to-Document Relationships

Every chunk needs to retain a link back to its source document and its position, both for display and for deduplication logic.

from dataclasses import dataclass


@dataclass
class Chunk:
    parent_doc_id: str
    chunk_index: int
    text: str


def chunk_document(doc_id: str, text: str, max_chunk_chars: int = 500) -> list[Chunk]:
    pieces = recursive_chunks(text, max_chunk_chars=max_chunk_chars)
    return [Chunk(parent_doc_id=doc_id, chunk_index=i, text=piece) for i, piece in enumerate(pieces)]


def test_chunk_document_preserves_order_and_parent():
    text = "First paragraph here.\n\nSecond paragraph here.\n\nThird paragraph here."
    chunks = chunk_document("doc-1", text, max_chunk_chars=40)
    assert all(c.parent_doc_id == "doc-1" for c in chunks)
    assert [c.chunk_index for c in chunks] == list(range(len(chunks)))
    print(f"PASS: chunk_document produced {len(chunks)} ordered chunks for doc-1")


test_chunk_document_preserves_order_and_parent()

Storing parent_doc_id and chunk_index on every chunk (as columns in the documents table from Lesson 6, or a related table) makes it possible to show a user which original document a result came from, to fetch neighboring chunks for additional context when a result is used in a RAG prompt, and to delete or re-index all chunks belonging to one source document without hunting for them by content.

Common Mistakes

• Splitting purely by character or token count with no regard for structure. As shown, this reliably cuts sentences and ideas in half, producing chunks that embed and read poorly.
• Using no overlap at all between adjacent chunks. This maximizes the chance that content straddling a boundary is not fully represented in any single chunk.
• Choosing one chunk size and never revisiting it. Chunk size is a tunable parameter with measurable effects on retrieval quality (Lesson 9); treating it as a fixed default rather than something to evaluate against real data leaves easy quality gains unclaimed.

Best Practices

• Default to structure-aware, recursive splitting over naive fixed-size splitting for anything beyond a quick prototype — it produces chunks that are both semantically coherent and cheap to implement, as shown above.
• Always store the parent document ID and chunk position alongside each chunk's text and embedding, so results can be traced back, displayed in context, and cleanly re-indexed.
• Tune chunk size and overlap using measured retrieval quality on representative queries, not intuition — the next lesson provides the methodology to do this rigorously.