Project 2: Build a Document Question-Answering System

This project builds a system that answers questions grounded in a specific set of documents rather than the model's general knowledge. The primary approach here uses the File Search tool covered in Unit 16, which handles chunking, embedding, and retrieval inside a managed vector store. The direct PDF-handling approach from Unit 7 is noted as an alternative at the end for cases where File Search's managed pipeline is not the right fit.

Scope and Design Decisions

The system ingests a folder of PDFs and text files — think internal policy documents, product manuals, or a knowledge base — and answers natural-language questions with citations back to the source documents. Three decisions define the shape of the implementation:

1. File Search over manual chunking. Building a custom chunking and embedding pipeline (Unit 20's approach) gives more control, but File Search's vector store handles document parsing, chunking, and retrieval as a managed service, which is the right trade-off when the corpus is standard document formats and the team does not need to tune the retrieval algorithm itself.
2. Citations are structured, not narrative. Instead of asking the model to mention its sources in prose, the response schema (Unit 6) forces every answer to include an explicit list of source file names, so the citation cannot be silently dropped by the model.
3. Unanswerable questions are a first-class outcome. The system must distinguish "the documents contain the answer" from "the documents don't cover this," rather than letting the model guess from general knowledge, which would defeat the purpose of grounding.

Building the Vector Store

from openai import OpenAI
from pathlib import Path

client = OpenAI()

def build_vector_store(name: str, document_dir: str) -> str:
    vector_store = client.vector_stores.create(name=name)

    file_ids = []
    for path in Path(document_dir).glob("*"):
        if path.suffix.lower() not in {".pdf", ".txt", ".md"}:
            continue
        with open(path, "rb") as f:
            uploaded = client.files.create(file=f, purpose="assistants")
        file_ids.append(uploaded.id)

    client.vector_stores.file_batches.create_and_poll(
        vector_store_id=vector_store.id,
        file_ids=file_ids,
    )
    return vector_store.id

This function does two distinct jobs, deliberately separated. First it uploads each raw file to get an OpenAI file ID — the file upload step is format-agnostic, so PDFs and plain text go through identically. Then it attaches those file IDs to a vector store in a single batch call. create_and_poll blocks until OpenAI finishes parsing and embedding every file in the batch, which matters because querying a vector store before indexing completes silently returns incomplete results rather than an error — polling to completion here avoids a subtle race condition in any code that runs immediately afterward.

Note: File Search chunking behavior and default chunk sizes are managed by OpenAI and can change between SDK versions. If retrieval quality on a specific corpus needs tuning, check current documentation for configurable chunking parameters before assuming the defaults are fixed.

Structured, Citation-Backed Answers

from pydantic import BaseModel

class SourcedAnswer(BaseModel):
    answer: str
    is_answerable_from_documents: bool
    source_files: list[str]

def ask_question(vector_store_id: str, question: str) -> SourcedAnswer:
    response = client.responses.parse(
        model="gpt-5.6-terra",
        input=[
            {
                "role": "system",
                "content": (
                    "Answer only using the attached documents. If the documents "
                    "do not contain the answer, set is_answerable_from_documents "
                    "to false and leave the answer generic."
                ),
            },
            {"role": "user", "content": question},
        ],
        tools=[{"type": "file_search", "vector_store_ids": [vector_store_id]}],
        text_format=SourcedAnswer,
    )
    return response.output_parsed

responses.parse combines tool use and structured output in one call, exactly as in Unit 6: the model can invoke file_search as many times as it needs while reasoning, and the final answer is still coerced into the SourcedAnswer schema before it reaches application code. The is_answerable_from_documents field is what actually enforces grounding — it gives the model an explicit place to admit "not in the documents" instead of quietly falling back to pretrained knowledge, which is the single most common failure mode of naive RAG systems built without this field.

source_files depends on the model correctly naming files it retrieved from, which is reliable for well-separated documents but can blur when several files cover overlapping topics. A stricter implementation would cross-reference response.output[i].content[j].annotations — the File Search tool attaches file citation annotations directly to the output text — and use those annotations as the source of truth instead of trusting the model's own listing in the schema.

def ask_question_with_verified_citations(vector_store_id: str, question: str) -> dict:
    response = client.responses.create(
        model="gpt-5.6-terra",
        input=[{"role": "user", "content": question}],
        tools=[{"type": "file_search", "vector_store_ids": [vector_store_id]}],
    )
    text_output = response.output_text
    cited_files = set()
    for item in response.output:
        if item.type != "message":
            continue
        for content in item.content:
            for annotation in getattr(content, "annotations", []):
                if annotation.type == "file_citation":
                    cited_files.add(annotation.filename)
    return {"answer": text_output, "verified_sources": sorted(cited_files)}

This second version trades the clean Pydantic schema for citations pulled directly from the API's own annotation metadata — a stronger guarantee, since the citation is generated by the retrieval system rather than the language model's free-text description of what it used. Production systems that need auditable sourcing (compliance, legal, medical) should prefer annotation-based citations over model-reported ones; systems where citations are a nice-to-have for user trust can use the simpler structured-output version.

Handling Document Updates

def replace_document(vector_store_id: str, old_filename: str, new_path: str) -> None:
    files = client.vector_stores.files.list(vector_store_id=vector_store_id)
    for f in files.data:
        file_info = client.files.retrieve(f.id)
        if file_info.filename == old_filename:
            client.vector_stores.files.delete(vector_store_id=vector_store_id, file_id=f.id)
            client.files.delete(f.id)

    with open(new_path, "rb") as fh:
        uploaded = client.files.create(file=fh, purpose="assistants")
    client.vector_stores.files.create_and_poll(
        vector_store_id=vector_store_id, file_id=uploaded.id
    )

Document sets change over time, and this is the operation most tutorials skip. Deleting the old vector store file entry and the underlying file object separately matters — removing only the vector store association leaves an orphaned file object accumulating storage cost, while removing only the file object can leave a stale reference in the vector store. Re-indexing after replacement, rather than trying to patch an existing embedding, is correct because embeddings are computed from full document content and cannot be partially updated.

Testing Without a Real Vector Store

class FakeParsedResponse:
    def __init__(self, parsed):
        self.output_parsed = parsed

def test_unanswerable_flag_is_respected(monkeypatch_parse):
    fake_answer = SourcedAnswer(
        answer="The provided documents do not cover this topic.",
        is_answerable_from_documents=False,
        source_files=[],
    )

    def fake_parse(**kwargs):
        return FakeParsedResponse(fake_answer)

    monkeypatch_parse(fake_parse)
    result = ask_question("fake-store-id", "What is the CEO's favorite color?")
    assert result.is_answerable_from_documents is False
    assert result.source_files == []
    print("PASS: unanswerable questions are flagged instead of guessed")

def _make_monkeypatch():
    original = client.responses.parse
    def apply(fn):
        client.responses.parse = fn
    def restore():
        client.responses.parse = original
    return apply, restore

monkeypatch_parse, restore_parse = _make_monkeypatch()
test_unanswerable_flag_is_respected(monkeypatch_parse)
restore_parse()

The test replaces client.responses.parse with a fake function returning a hand-built SourcedAnswer, which lets ask_question's logic be verified — that it returns exactly what the API layer gives it — without a network call, a real vector store, or real documents. This is the same substitution-based testing pattern as earlier units: swap the SDK boundary, not the business logic.

Alternative Approach: Direct PDF Extraction

For a small, static set of PDFs where full control over chunking matters more than convenience — for example, tables that need custom parsing — the Unit 7 approach of extracting text directly (with pypdf or a similar library) and feeding relevant excerpts into the prompt as context remains valid, especially combined with the manual embedding-based retrieval from Unit 20 for corpora too large to fit in a single context window. File Search is preferred by default in this project because it removes the operational burden of running that pipeline yourself.

Extending This Project

Add per-user access control by creating one vector store per permission tier and selecting which store to query based on the requester's role, and add a feedback loop that logs which citations users click through to, which is valuable data for identifying documents that are frequently retrieved but poorly worded.

Common Mistakes

• Querying a vector store immediately after starting file uploads. Indexing is asynchronous; querying before it completes returns partial or empty results. Always poll to completion with create_and_poll before serving queries.
• Trusting model-reported source files over API-provided citation annotations. The model can misname or omit sources in free text; annotation objects from the File Search tool are a stronger source of truth for anything audit-sensitive.
• Deleting only the vector store association when removing a document. This leaves an orphaned file object in storage. Delete both the vector store file link and the underlying file object.

Best Practices

• Give the model an explicit way to say "not in the documents." A boolean or enum field for answerability is what actually prevents hallucinated answers dressed up as grounded ones.
• Re-index rather than patch when documents change. Embeddings are computed over full content; there is no safe partial update.
• Match the retrieval approach to corpus size and control needs. File Search for standard documents at moderate scale; custom embedding pipelines when retrieval tuning or non-standard formats are required.