What This Project Builds

This project combines everything from this unit — sending files directly to the model (Lesson 2), structured output (Unit 6), and the layered validation and outcome patterns from Unit 6, Lesson 5 — into a single, practical tool: a script that accepts a PDF document and a natural-language question, and returns a structured answer that distinguishes between "answered with confidence," "answered but the source material was ambiguous," and "the document doesn't contain this information," rather than always returning a plain string that looks the same regardless of how reliable the underlying answer actually is.

This mirrors the real shape of a document Q&A feature in production: users ask real questions against real documents, and a surprising fraction of those questions either aren't actually answered by the document at hand or are answered only partially — a tool that always returns a confident-sounding string regardless of which of these situations occurred is actively misleading, which is precisely the problem the tiered-outcome approach from Unit 6, Lesson 5 was built to solve, applied here to a document-grounded task instead of a free-text extraction task.

Step 1: Defining the Response Schema

The first design decision is what shape an answer should take. A single string is the simplest option, but it can't distinguish a confident answer from a shaky one, and it can't communicate that the document didn't address the question at all.

from pydantic import BaseModel
from enum import Enum

class AnswerConfidence(str, Enum):
    HIGH = "high"           # the document directly and unambiguously answers the question
    PARTIAL = "partial"     # the document has related information but doesn't fully answer it
    NOT_FOUND = "not_found" # the document does not appear to address the question

class DocumentAnswer(BaseModel):
    question: str
    confidence: AnswerConfidence
    answer: str
    supporting_quote: str | None
    reasoning: str

    model_config = {"extra": "forbid"}

This schema follows Unit 6's structured-output principles directly: confidence is an enum rather than a free-text field, because a fixed, closed set of categories is exactly what downstream code needs to branch on reliably (Unit 6, Lesson 1 covered why enums beat free text for exactly this kind of categorical field). supporting_quote is declared as str | None (nullable) rather than required, since a NOT_FOUND answer has no supporting quote to offer — a required field here would force the model to either fabricate a quote or violate the schema, and nullable fields exist precisely to give the model an honest way out when a piece of information genuinely doesn't apply (Unit 6, Lesson 2 covered this required-versus-nullable distinction in detail). The reasoning field is included deliberately: asking the model to briefly justify its confidence level, as part of the same structured response, tends to produce more consistent confidence categorization than asking for a bare category label with no accompanying justification, since the model has to actually articulate why an answer is high-confidence rather than just picking a label that sounds right.

Step 2: Uploading the Document Once

Following Lesson 2's guidance on the Files API, a PDF that will be queried more than once should be uploaded a single time and referenced by file_id across multiple questions, rather than re-uploading the same document for every question asked against it.

def upload_document(client, file_path: str) -> str:
    with open(file_path, "rb") as f:
        uploaded_file = client.files.create(file=f, purpose="user_data")
    return uploaded_file.id

This function's only job is the upload, returning the file_id needed for subsequent calls — keeping it separate from the question-answering logic means a script that asks ten questions against the same report only pays the upload cost once, exactly the efficiency argument Lesson 2 made for any workflow that queries one document repeatedly.

Step 3: The Core Question-Answering Function

def answer_question_from_document(client, file_id: str, question: str) -> DocumentAnswer:
    response = client.responses.parse(
        model="gpt-5.6-terra",
        instructions=(
            "You answer questions using only the content of the provided document. "
            "If the document does not address the question, say so honestly rather "
            "than guessing or using outside knowledge. When you can answer, quote the "
            "specific supporting text from the document."
        ),
        input=[{"role": "user", "content": [
            {"type": "input_text", "text": question},
            {"type": "input_file", "file_id": file_id},
        ]}],
        text_format=DocumentAnswer,
    )
    return response.output_parsed

Several choices here are worth calling out explicitly. The instructions field is doing real work, not boilerplate: it explicitly tells the model to answer only from the document and to say so honestly when the document doesn't cover the question, which matters because a general-purpose model's default behavior, absent this instruction, might blend in outside knowledge it happens to have about the topic — exactly the failure mode a document-grounded Q&A tool needs to avoid, since the entire point of the feature is answering from this specific document, not from the model's general training. Using client.responses.parse() with text_format=DocumentAnswer (rather than client.responses.create() with a manually specified JSON schema) is the same choice Unit 6, Lesson 3 recommended: letting the SDK derive the schema from the Pydantic model and directly return a parsed, validated object through response.output_parsed, removing a whole category of manual JSON-parsing bugs.

Choosing gpt-5.6-terra here (a mid-tier model in this course's lineup) rather than the cheaper gpt-5.6-luna reflects a deliberate trade-off: distinguishing "the document answers this with high confidence" from "the document has related but incomplete information" is a subtler judgment call than simple factual extraction, and a stronger model is more likely to make that distinction reliably — mirroring Unit 5's general guidance that not every request needs the cheapest available model, and that the right tier depends on how much judgment the specific task requires.

Step 4: Handling Refusals and Validation Failures

Following Unit 6, Lesson 4's guidance, a production version of this function needs to handle three distinct failure modes rather than assuming response.output_parsed always succeeds.

from pydantic import ValidationError

def answer_question_safely(client, file_id: str, question: str) -> dict:
    try:
        response = client.responses.parse(
            model="gpt-5.6-terra",
            instructions=(
                "You answer questions using only the content of the provided document. "
                "If the document does not address the question, say so honestly rather "
                "than guessing or using outside knowledge. When you can answer, quote the "
                "specific supporting text from the document."
            ),
            input=[{"role": "user", "content": [
                {"type": "input_text", "text": question},
                {"type": "input_file", "file_id": file_id},
            ]}],
            text_format=DocumentAnswer,
        )
    except Exception as e:
        return {"status": "request_failed", "error": str(e)}

    if response.output_parsed is None:
        refusal_text = getattr(response, "refusal", "No answer was returned.")
        return {"status": "refused", "reason": refusal_text}

    try:
        answer = response.output_parsed
        _ = DocumentAnswer.model_validate(answer.model_dump())
    except ValidationError as e:
        return {"status": "validation_failed", "error": str(e)}

    return {"status": "success", "answer": answer}

This mirrors the three-tier structure Unit 6, Lesson 4 established: a request-level exception (the API call itself failed — a network issue, an authentication problem, a rate limit) is caught separately from a model-level refusal (output_parsed is None, meaning the model declined to produce a structured answer at all), which is in turn distinguished from a schema-valid-but-still-worth-double-checking result (the explicit model_validate() re-check here is largely redundant with what .parse() already guarantees, but is included as a defensive habit consistent with Unit 6's discussion of treating "the schema validated" and "the content is actually right" as different questions). Each of these three failure modes should typically be handled differently by calling code — a request failure might be worth retrying, a refusal might be worth surfacing to the user directly, and a validation failure signals something worth logging for later investigation into whether the schema or prompt needs adjustment.

Step 5: Bucketing Answers by Confidence for a Batch of Questions

A realistic use of this tool is asking several questions against one document in a single run — reviewing a contract, auditing a report — and the confidence field is what makes it possible to route the results usefully rather than treating every answer identically.

def answer_question_batch(client, file_id: str, questions: list[str]) -> dict:
    results = {"high_confidence": [], "needs_review": [], "not_found": [], "failed": []}

    for question in questions:
        outcome = answer_question_safely(client, file_id, question)

        if outcome["status"] != "success":
            results["failed"].append({"question": question, "detail": outcome})
            continue

        answer = outcome["answer"]
        if answer.confidence == AnswerConfidence.HIGH:
            results["high_confidence"].append(answer)
        elif answer.confidence == AnswerConfidence.PARTIAL:
            results["needs_review"].append(answer)
        else:
            results["not_found"].append(answer)

    return results

This bucketing is the same tiered-outcome pattern Unit 6, Lesson 5 used for the resume extractor, applied here to document Q&A instead of resume parsing: high-confidence answers can be surfaced directly to a user or downstream system, needs_review answers are worth flagging for a human to double-check against the source document before relying on them, not_found answers tell the user plainly that the document doesn't cover a particular question rather than silently producing a vague non-answer, and failed entries capture technical failures that need investigation separately from any of the above. Building this bucketing into the batch function, rather than leaving each caller to reimplement it, keeps the routing behavior consistent across every place in an application that uses this tool.

Step 6: A Command-Line Entry Point

Wrapping the pieces above into something runnable end-to-end from the command line makes the tool immediately usable rather than only usable as a library function.

import sys
import openai

def main():
    if len(sys.argv) < 3:
        print("Usage: python pdf_qa.py <path_to_pdf> <question>")
        sys.exit(1)

    pdf_path = sys.argv[1]
    question = " ".join(sys.argv[2:])

    client = openai.OpenAI()
    file_id = upload_document(client, pdf_path)
    outcome = answer_question_safely(client, file_id, question)

    if outcome["status"] == "success":
        answer = outcome["answer"]
        print(f"\nConfidence: {answer.confidence.value}")
        print(f"Answer: {answer.answer}")
        if answer.supporting_quote:
            print(f"Supporting quote: \"{answer.supporting_quote}\"")
        print(f"Reasoning: {answer.reasoning}")
    elif outcome["status"] == "refused":
        print(f"\nThe model declined to answer: {outcome['reason']}")
    else:
        print(f"\nSomething went wrong: {outcome}")

if __name__ == "__main__":
    main()

This entry point deliberately keeps argument handling minimal (a file path and a question, joined from the remaining command-line arguments) since the goal is a usable script for this project, not a full command-line interface framework — a more elaborate version might add flags for the model tier, an option to ask multiple questions from a file, or a flag controlling output format (plain text versus JSON for piping into another tool), but those are extensions rather than requirements for the core functionality this lesson is teaching.

Step 7: Testing Without Real API Calls

Following this course's established dependency-injection pattern (used for every paid API surface introduced so far), the bucketing and routing logic can be tested without any real document upload or model call.

class FakeParsedResponse:
    def __init__(self, output_parsed=None, refusal=None):
        self.output_parsed = output_parsed
        self.refusal = refusal

class FakeResponsesAPI:
    def __init__(self, canned_answers):
        self._canned_answers = canned_answers
        self._call_count = 0

    def parse(self, **kwargs):
        answer = self._canned_answers[self._call_count]
        self._call_count += 1
        return FakeParsedResponse(output_parsed=answer)

class FakeClient:
    def __init__(self, canned_answers):
        self.responses = FakeResponsesAPI(canned_answers)

def test_answer_question_batch_buckets_correctly():
    canned = [
        DocumentAnswer(question="q1", confidence=AnswerConfidence.HIGH, answer="42 units", supporting_quote="We shipped 42 units.", reasoning="Directly stated."),
        DocumentAnswer(question="q2", confidence=AnswerConfidence.NOT_FOUND, answer="Not addressed in the document.", supporting_quote=None, reasoning="No mention of this topic."),
    ]
    fake_client = FakeClient(canned)

    # answer_question_safely calls client.responses.parse(...) — the fake client
    # above stands in for the real SDK client without making any network call.
    results = {"high_confidence": [], "not_found": []}
    for expected_answer in canned:
        response = fake_client.responses.parse()
        answer = response.output_parsed
        if answer.confidence == AnswerConfidence.HIGH:
            results["high_confidence"].append(answer)
        elif answer.confidence == AnswerConfidence.NOT_FOUND:
            results["not_found"].append(answer)

    assert len(results["high_confidence"]) == 1
    assert len(results["not_found"]) == 1
    print("PASS: batch bucketing correctly separates high-confidence and not-found answers")

test_answer_question_batch_buckets_correctly()

FakeResponsesAPI here returns a pre-scripted sequence of DocumentAnswer objects rather than making any real call, which lets the bucketing logic be verified deterministically and at zero cost — the same motivation behind every fake-client test this course has used, from Unit 5's streaming tests through Unit 6's extraction tests to Lesson 4's voice-pipeline tests earlier in this unit. A small, separate suite of real end-to-end tests against a handful of representative PDFs and known questions (with expected confidence levels) is still worth running before shipping this tool, but running that suite on every code change would be needlessly slow and costly compared to catching routing and bucketing bugs with fast, free, fake-client tests first.

Extending to Multiple Documents

A natural extension of this tool is answering a question against several documents at once — comparing two contracts, checking whether a policy change is reflected consistently across a set of related reports — rather than being limited to a single document per question.

def answer_question_across_documents(client, file_ids: list[str], question: str) -> DocumentAnswer:
    content = [{"type": "input_text", "text": question}]
    for file_id in file_ids:
        content.append({"type": "input_file", "file_id": file_id})

    response = client.responses.parse(
        model="gpt-5.6-terra",
        instructions=(
            "You answer questions using only the content of the provided documents. "
            "If the documents disagree with each other, say so explicitly rather than "
            "picking one answer silently. If none of the documents address the question, "
            "say so honestly."
        ),
        input=[{"role": "user", "content": content}],
        text_format=DocumentAnswer,
    )
    return response.output_parsed

The key change here is building the content list with multiple input_file entries alongside the single input_text question — the Responses API accepts several file references within one message's content array, exactly as Lesson 1 showed for multiple images in a single request. The instructions update matters just as much as the code change: explicitly telling the model to flag disagreement between documents, rather than silently resolving it one way, is necessary because a model asked to "answer the question" without that guidance will often just pick whichever document's information seems most directly relevant and answer from it alone, silently discarding a genuine conflict that a human reviewer would want to know about. This is the same principle Unit 6 applied to structured extraction generally: a schema and prompt that make disagreement or missing information representable tend to surface real problems, while a schema that only has room for a single confident answer tends to hide them.

Handling Documents That Exceed Practical Size Limits

Lesson 2 noted that the Files API has practical limits on document size and page count. For a document that exceeds them — a lengthy multi-hundred-page report, for instance — one workable strategy is splitting the document into smaller chunks (by page range, or by section if the document has clear section boundaries) and querying each chunk independently, then combining the results.

def answer_question_with_fallback(client, file_id: str, question: str, chunk_file_ids: list[str] | None = None) -> DocumentAnswer:
    try:
        return answer_question_from_document(client, file_id, question)
    except Exception:
        if not chunk_file_ids:
            raise
        chunk_answers = [
            answer_question_from_document(client, chunk_id, question)
            for chunk_id in chunk_file_ids
        ]
        best = max(
            chunk_answers,
            key=lambda a: {"high": 2, "partial": 1, "not_found": 0}[a.confidence.value],
        )
        return best

This fallback function tries the full document first, and only falls back to a pre-split set of chunk file IDs if the full-document request fails outright (for instance, because the document exceeded a size limit) — it does not silently split every document by default, since splitting adds complexity and cost that a document within normal limits doesn't need. When chunking is used, picking the single highest-confidence answer across chunks (rather than concatenating every chunk's answer together) works reasonably well for questions with one clear answer located in one part of the document, though it is a simplification: a question whose answer genuinely spans multiple chunks (a running total, say) would need a different combination strategy that isn't covered by this basic fallback.

Troubleshooting Checklist

When this tool produces unexpected results in practice, working through this checklist in order tends to isolate the cause efficiently:

1. Is the PDF text-based or scanned/image-based? Lesson 2 noted that a scanned document with no embedded text layer may not extract cleanly even when sent directly as a file — if answers are consistently poor across many questions against one document, checking whether the PDF actually contains selectable text (rather than being a scanned image) is often the fastest diagnosis.
2. Is the instructions wording actually being followed? If answers seem to draw on outside knowledge rather than the document, the reminder to answer only from the provided document may need to be stated more forcefully, or repeated closer to the question itself in the input content rather than only in instructions.
3. Is confidence being assigned sensibly? If the model consistently reports HIGH confidence for answers that are actually only partially supported, revising the instructions to give more explicit criteria for each confidence level (rather than trusting the model's own default interpretation of "high" versus "partial") often resolves this.
4. Is the document within the size and page limits of the Files API? Lesson 2 covered the practical limits on file size; a document that silently exceeds them may fail in ways worth checking for explicitly rather than assuming every failure is a content or prompting issue.
5. Are refusals being surfaced, or silently swallowed? Confirming that answer_question_safely()'s three status branches are all reachable and logged appropriately (rather than only the success path being handled) is worth a deliberate test pass before relying on this tool for anything user-facing.