Why Document Quality Is the Highest-Leverage Fix

When a file-search-backed assistant gives a wrong or vague answer, the instinctive response is often to tweak the prompt or switch models. In practice, the single highest-leverage place to look first is the input documents themselves — because retrieval can only surface chunks that exist and are well-formed. No amount of prompt engineering compensates for a chunk that split a sentence in half, or a scanned PDF with no extractable text at all. This lesson covers the concrete, practical steps that improve what actually gets indexed, building on the ingestion mechanics from Lesson 3.

Problem 1: Documents With No Real Text Layer

A PDF that was produced by scanning a paper document is, from a text-extraction perspective, just an image. There is no embedded text layer for file_search's ingestion pipeline to extract, so the resulting chunks are empty or nonexistent, and that document is effectively invisible to retrieval even though it "uploaded successfully."

The fix is to run OCR (optical character recognition) on such documents before uploading them, producing a text layer that extraction can actually find. Unit 7 already covers PDF handling in depth, including OCR workflows for scanned documents — apply that same preprocessing here before ingestion, not after discovering that a document never surfaces in any search.

def looks_extractable(pdf_text, min_chars_per_page, page_count):
    if page_count == 0:
        return False
    average_chars = len(pdf_text) / page_count
    return average_chars >= min_chars_per_page


def test_looks_extractable_flags_scanned_pdf():
    # A scanned PDF with no OCR often yields little to no extracted text.
    near_empty_text = " " * 20
    assert looks_extractable(near_empty_text, min_chars_per_page=200, page_count=5) is False
    print("PASS: near-empty extracted text is flagged as not extractable")


def test_looks_extractable_accepts_normal_document():
    normal_text = "This is a paragraph. " * 300
    assert looks_extractable(normal_text, min_chars_per_page=200, page_count=5) is True
    print("PASS: normal text-dense document passes the extractability check")


test_looks_extractable_flags_scanned_pdf()
test_looks_extractable_accepts_normal_document()

This small heuristic function checks whether a PDF likely has a usable text layer by comparing the average number of extracted characters per page against a threshold — a scanned document with no OCR will extract to almost nothing, while a normal text document extracts to a substantial amount of text per page. Running a check like this as a pre-ingestion gate (using a local PDF text extraction library, separate from the OpenAI API itself) lets you catch and route scanned documents to an OCR step automatically, instead of silently ingesting documents that will never contribute a single relevant chunk.

Problem 2: Poor Document Structure

Chunking algorithms rely on structural signals — headings, paragraph breaks, list formatting — to decide where one topical unit ends and another begins. A document that is one enormous unbroken wall of text (common in poorly-formatted Word-to-PDF exports) gives the chunker nothing to work with, often producing chunks that split mid-topic or even mid-sentence.

Improving structure before upload, where you have control over the source document, pays off directly:

• Use real headings (not just bold, larger text that looks like a heading visually but carries no structural markup) when the source format supports it.
• Break dense paragraphs into shorter ones organized around a single idea each.
• Use actual list formatting for enumerated content instead of run-on sentences separated by commas.

For content you author directly — internal wikis exported to Markdown, for instance — this is entirely within your control and costs little at authoring time compared to the retrieval quality it buys later.

Problem 3: Boilerplate and Repeated Noise

Headers, footers, page numbers, and repeated legal disclaimers that appear on every page of a PDF get extracted along with the real content, and they can pollute chunks — especially short chunks where boilerplate might make up a large fraction of the chunk's total content, diluting the embedding's semantic signal.

A simple mitigation is stripping known boilerplate patterns from extracted text before it's uploaded, when you're able to pre-process a document yourself rather than relying purely on the automatic pipeline:

import re


def strip_boilerplate(text, patterns):
    cleaned = text
    for pattern in patterns:
        cleaned = re.sub(pattern, "", cleaned, flags=re.MULTILINE)
    return cleaned.strip()


boilerplate_patterns = [
    r"^Confidential — Internal Use Only$",
    r"^Page \d+ of \d+$",
]

raw_text = (
    "Confidential — Internal Use Only\n"
    "Section 3: Data Retention Policy\n"
    "All customer data is retained for 90 days.\n"
    "Page 4 of 12"
)

cleaned_text = strip_boilerplate(raw_text, boilerplate_patterns)
print(cleaned_text)

This function applies a list of regular expression patterns to remove known repeated boilerplate lines from extracted text, leaving only the substantive content. The patterns list is deliberately passed in as an argument rather than hardcoded, since boilerplate differs across document sets — a legal team's disclaimers look nothing like an engineering wiki's footer — and keeping the function generic makes it reusable across different ingestion pipelines. This kind of cleaning step happens on text you control before upload, not on documents already ingested by the hosted pipeline, since file_search does not provide a way to edit already-indexed chunk text directly.

Problem 4: Tables and Structured Data

Tables are one of the most reliable sources of retrieval failure, because the meaning of a table cell depends on its row and column headers, and naive text extraction can flatten a table into a sequence of numbers with the header context stripped away or pushed far from the values it describes, especially if a table spans a chunk boundary.

Where you control document creation, converting critical tables into a more retrieval-friendly textual form — restating each row as a full sentence — often improves retrieval far more than any downstream tuning:

def table_row_to_sentence(row, headers):
    parts = [f"{header} is {value}" for header, value in zip(headers, row)]
    return ", ".join(parts) + "."


headers = ["Plan", "Monthly Price", "Support Level"]
rows = [
    ["Basic", "$9", "Email only"],
    ["Pro", "$29", "Priority chat and email"],
    ["Enterprise", "$99", "Dedicated account manager"],
]

sentences = [table_row_to_sentence(row, headers) for row in rows]
for sentence in sentences:
    print(sentence)

This produces output like Plan is Basic, Monthly Price is $9, Support Level is Email only. — verbose compared to a compact table, but far more robust to chunking, because each row now carries its own header context inline rather than depending on being read alongside a separate header row that a chunk boundary might separate it from. This transformation is worth applying selectively to the tables that matter most for question-answering (pricing tables, comparison charts, policy matrices), not universally, since it does increase token count and can be unnecessary for tables that aren't likely to be queried directly.

Problem 5: Outdated or Conflicting Content

A vector store accumulates documents over time, and if an old version of a policy is never removed when a new version is uploaded, retrieval can surface either version — or both — leaving the model to reconcile contradictory "facts" that are really just stale content that should have been retired.

The fix here isn't a document preparation technique so much as a process discipline: treat document replacement as a delete-and-reupload operation, not an add-only operation.

def replace_document(client, vector_store_id, old_file_id, new_file_path, attributes):
    client.vector_stores.files.delete(
        vector_store_id=vector_store_id,
        file_id=old_file_id,
    )

    with open(new_file_path, "rb") as f:
        result = client.vector_stores.files.upload_and_poll(
            vector_store_id=vector_store_id,
            file=f,
            attributes=attributes,
        )

    return result

Note: The exact deletion method name and signature under vector_stores.files are version-specific — confirm against current documentation before relying on this pattern in production.

This function removes the outdated file from the vector store first, then uploads the replacement, which ensures there is never a moment where both the stale and the current version coexist and compete in retrieval. Tracking a mapping from logical document identity (like "refund policy") to its current file ID — mentioned in Lesson 3 as a best practice — is what makes calling this function correctly possible; without that record, you'd have to search the vector store's file list to find the old version first.

Measuring Whether Preparation Actually Helped

Improvements to document preparation should be validated, not assumed. A lightweight approach is maintaining a small fixed set of test questions with known correct source documents, and checking whether the right file shows up in citations after a change:

def evaluate_retrieval(ask_question_fn, store_id, test_cases):
    passed = 0
    for question, expected_source in test_cases:
        _, sources = ask_question_fn(store_id, question)
        if expected_source in sources:
            passed += 1
        else:
            print(f"MISS: '{question}' expected {expected_source}, got {sources}")

    print(f"{passed}/{len(test_cases)} retrieval checks passed")

This function runs a fixed list of (question, expected_source_file) pairs against a real or fake ask_question function and reports how many retrieved the expected source document among their citations. Running this same fixed test set before and after a document preparation change (splitting badly structured documents, stripping boilerplate, converting key tables) gives you an objective before-and-after comparison, rather than a subjective impression that retrieval "seems better."

Common Mistakes

Uploading scanned PDFs without OCR and assuming a successful upload status means the content is searchable, when in fact a document with no text layer contributes nothing to retrieval regardless of upload status — always verify extractability before trusting a document is actually indexed.

Leaving stale document versions in a vector store after uploading an update, which lets outdated and current information compete in retrieval and produces inconsistent answers — always pair an update with removal of the version it replaces.

Assuming a preparation change helped without measuring it, relying on spot-checking a few questions informally rather than a consistent test set — maintain a small fixed evaluation set and compare results objectively before and after any change.

Best Practices

Run an extractability and structure check on every document before ingestion, catching scanned PDFs, near-empty extractions, and severely unstructured documents before they enter the vector store rather than after a user reports a missing answer.

Convert high-value tables to sentence form when precision on tabular facts matters, accepting the extra verbosity in exchange for reliable retrieval of specific figures.

Maintain a small, fixed retrieval evaluation set per vector store, and re-run it after any meaningful change to documents, chunking, or metadata, so quality claims are always backed by a repeatable check.