Preparing Text for Embedding

Text Preparation for Embedding

An embedding model turns whatever text it is given into a vector — it does not know or care whether that text is clean prose or a mess of HTML tags, repeated whitespace, and boilerplate navigation links scraped along with the real content. The quality of a semantic search system is bounded by the quality of the text that goes into the embedding call, which makes text preparation one of the highest-leverage steps in the whole pipeline, and one that is easy to skip when a demo works on tidy sample data and then quietly underperforms on real data.

Why Raw Text Often Produces Poor Embeddings

Three problems show up repeatedly with real-world text sources (scraped web pages, exported PDFs, database fields written by humans over years):

1. Boilerplate dilutes meaning. A support article's true content might be 200 words, but if it is scraped with a 150-word navigation menu, cookie notice, and footer attached, a large fraction of the text that gets embedded has nothing to do with the article's actual topic. The resulting vector partially represents "generic website chrome" rather than the article's subject.
2. Formatting noise adds no semantic value but changes the input. Excess whitespace, HTML tags, markdown symbols, and control characters are not stripped automatically by the embedding model — they are tokenized like any other characters, spending part of the input on symbols that carry no meaning.
3. Inconsistent preprocessing between indexing time and query time breaks comparisons. If documents are lowercased and stripped of punctuation before embedding, but user queries are embedded raw, the two are not being compared on equal footing, and any preprocessing-related distortion adds noise rather than being consistent enough to cancel out.

A Basic Cleaning Pipeline

Start with the transformations that are almost always safe: removing markup, collapsing whitespace, and trimming boilerplate patterns known ahead of time.

import re

def strip_html(text: str) -> str:
    """Remove HTML tags, leaving the text content."""
    return re.sub(r"<[^>]+>", " ", text)

def collapse_whitespace(text: str) -> str:
    """Collapse runs of whitespace (including newlines) into single spaces."""
    return re.sub(r"\s+", " ", text).strip()

def clean_text(text: str) -> str:
    """Apply the basic cleaning pipeline in a fixed, documented order."""
    text = strip_html(text)
    text = collapse_whitespace(text)
    return text

raw = """
<div class="article">
  <h1>Return Policy</h1>
  <p>Items   can be returned      within 30 days
  of purchase.</p>
</div>
"""
print(clean_text(raw))
# "Return Policy Items can be returned within 30 days of purchase."

strip_html uses a regular expression to remove anything between angle brackets — this is a pragmatic tool for well-formed HTML fragments, not a full HTML parser, so it can misbehave on malformed markup or text that legitimately contains a < character (like if x < 10). For content scraped from real web pages, a proper HTML parser (such as BeautifulSoup, covered as a dependency in earlier units) is more robust than a regular expression and should be preferred when available; the regex version here is shown because it has no external dependency and illustrates the idea clearly. collapse_whitespace matters because inconsistent spacing and line breaks add tokens that carry no semantic content, and normalizing them makes near-identical documents produce more consistent embeddings.

Deciding What to Normalize — and What Not To

Not every normalization that seems reasonable actually helps. This is where preparing text for embeddings differs from preparing text for older techniques like keyword search.

Why does lowercasing and stopword removal, which helped older keyword-based systems, often hurt here? Classic keyword search (TF-IDF, BM25) counts literal word overlap, so normalizing case and removing very common words reduces noise in that literal counting. Embedding models work completely differently: they are trained on large amounts of natural text and learn contextual meaning, including the meaning carried by function words and capitalization. Applying keyword-search-era normalization to embedding input removes signal the model was trained to use, rather than removing noise.

Deduplication Before Embedding

Embedding the same or near-identical text multiple times wastes API cost and, more importantly, can bias a search index by making one underlying piece of content appear to dominate simply because it exists in more copies.

def deduplicate_exact(texts: list[str]) -> list[str]:
    """Remove exact duplicate strings, preserving first-seen order."""
    seen = set()
    unique = []
    for text in texts:
        key = collapse_whitespace(text).lower()
        if key not in seen:
            seen.add(key)
            unique.append(text)
    return unique

def test_deduplicate_exact():
    texts = [
        "Return policy for electronics.",
        "Return  policy for electronics.",   # extra whitespace, otherwise identical
        "Shipping times vary by region.",
    ]
    result = deduplicate_exact(texts)
    assert len(result) == 2
    print("PASS: deduplicate_exact collapses whitespace-only duplicates")

test_deduplicate_exact()

This function normalizes each text (whitespace-collapsed, lowercased) only to build a comparison key for deduplication — it does not alter the text that actually gets embedded, which preserves the "don't force lowercase into the model input" guidance above while still catching duplicates that differ only in casing or spacing. Exact deduplication like this catches copy-pasted or re-scraped content; catching near-duplicates (two paragraphs that say the same thing in different words) requires comparing embeddings themselves, which is what Lesson 4's similarity techniques and the duplicate-detection ideas in Lesson 5 are for — that is a semantic problem, not a text-cleaning one.

Handling Length: Truncation and the Case for Chunking

Every embedding model has a maximum input length measured in tokens (not characters or words). Text longer than that limit needs to be handled deliberately.

def truncate_to_char_budget(text: str, max_chars: int) -> str:
    """A rough, dependency-free length guard.

    This is a coarse approximation — tokens and characters are not
    the same unit — but it prevents obviously oversized inputs from
    reaching the API at all. A tokenizer-based check is more precise
    when precision matters (see Unit 12 for tokenization details).
    """
    if len(text) <= max_chars:
        return text
    return text[:max_chars].rsplit(" ", 1)[0]

Truncating a long document to fit a length limit is a blunt tool: it silently discards everything past the cutoff, which may include the most relevant part of the document. For any document long enough to risk truncation, the better approach is chunking — splitting it into smaller, independently embedded pieces so that no information is thrown away and a search query can match the specific chunk that is actually relevant. Lesson 8 in this unit is dedicated entirely to chunking strategy; this lesson's truncation helper is a safety net for the rare oversized input that slips through, not a substitute for chunking a genuinely long document.

Common Mistakes

• Applying keyword-search preprocessing (lowercasing, stopword removal) to embedding input. This was covered above: it removes signal that modern embedding models rely on rather than removing noise.
• Cleaning documents at index time but not queries at search time (or vice versa). If HTML stripping and whitespace collapsing are applied to stored documents but the live user query is embedded raw, the two are prepared inconsistently, which adds avoidable noise to every comparison.
• Ignoring boilerplate in scraped or exported content. Repeated navigation text, footers, and disclaimers embedded alongside real content dilute the vector's meaning and can cause unrelated documents to appear similar simply because they share the same boilerplate.

Best Practices

• Apply the exact same cleaning function to documents and queries. Put the cleaning logic in one shared function (like clean_text above) and call it from both the indexing path and the query path — never duplicate the logic in two places where it can drift out of sync.
• Prefer chunking over truncation for long documents. Truncation discards information; chunking preserves it while keeping each piece within the model's input limit.
• Deduplicate before embedding, not after. Removing duplicates before the API call saves cost directly, rather than embedding the duplicate and discarding the result afterward.