Two-Step vs. One-Step Uploads

Getting a document into a vector store is conceptually two separate operations: uploading the raw file to OpenAI's file storage, and attaching that file to a specific vector store so it gets chunked, embedded, and indexed. The SDK gives you both a low-level path that makes this explicit and a convenience path that does both at once.

The Explicit Two-Step Path

from openai import OpenAI

client = OpenAI()

with open("employee-handbook.pdf", "rb") as f:
    uploaded_file = client.files.create(
        file=f,
        purpose="assistants",
    )

print(uploaded_file.id)

attachment = client.vector_stores.files.create(
    vector_store_id="vs_68f2a1c9e4b8...",
    file_id=uploaded_file.id,
)

print(attachment.status)

Note: The required purpose value for files destined for vector stores, and the exact method names under client.vector_stores.files, can change between SDK versions — confirm both against current OpenAI documentation before relying on them in production.

Walking through this: client.files.create uploads the raw bytes of employee-handbook.pdf to OpenAI's file storage and returns a File object with its own ID (separate from any vector store ID). At this point the file exists on OpenAI's servers but is not part of any vector store and cannot be searched yet. The second call, client.vector_stores.files.create, takes that already-uploaded file and attaches it to a specific vector store, which triggers the actual chunking and embedding pipeline. The returned attachment.status tells you whether that processing has completed, is in progress, or failed — the same asynchronous behavior described in Lesson 2.

This two-step path matters when you want to reuse the same uploaded file across multiple vector stores (a legal policy document that belongs in both a "legal" store and a "employee onboarding" store, for instance) without uploading the bytes twice.

The Convenience One-Step Path

For the common case — one file, one destination store, uploaded and attached together — the SDK provides a helper that collapses both steps:

with open("employee-handbook.pdf", "rb") as f:
    result = client.vector_stores.files.upload_and_poll(
        vector_store_id="vs_68f2a1c9e4b8...",
        file=f,
    )

print(result.status)

This does the same two operations as above, but also polls the vector store attachment until it reaches a terminal state (completed or failed) before returning, instead of returning immediately with a status of in_progress. That polling behavior is exactly what you want in a script that uploads a document and then needs to know, synchronously, whether it succeeded — you avoid writing your own retry loop.

Uploading Multiple Files in Batch

Uploading documents one at a time in a loop works, but for anything beyond a handful of files, a batch upload is both faster and easier to monitor:

import glob

file_paths = glob.glob("docs/*.pdf")
file_streams = [open(path, "rb") for path in file_paths]

batch = client.vector_stores.file_batches.upload_and_poll(
    vector_store_id="vs_68f2a1c9e4b8...",
    files=file_streams,
)

print(batch.status)
print(batch.file_counts.completed, "of", batch.file_counts.total, "succeeded")

for stream in file_streams:
    stream.close()

Note: Method names under file_batches, batch size limits, and supported file types are version-specific — verify current limits and supported formats in the official documentation before building a production ingestion pipeline around them.

This example collects every PDF in a local docs/ directory, opens each as a binary file stream, and submits them as a single batch to the vector store, then waits for the whole batch to finish processing. Batching matters for two practical reasons: it reduces the number of round trips to the API compared to uploading files one by one, and batch.file_counts gives you an aggregate view of how many files in the batch succeeded versus failed, which is far more useful for a bulk ingestion job than checking each file's status individually. Closing each file stream afterward is good hygiene — leaving many open file handles in a long-running ingestion process is an easy way to hit operating-system file-descriptor limits.

What Happens to a Document Behind the Scenes

Once a file is attached to a vector store, several things happen automatically that are worth understanding, because they explain what you can and cannot control at upload time:

1. Text extraction. For formats like PDF or DOCX, the text content is extracted from the document, including handling multi-column layouts and, in supported cases, text within simple tables.
2. Chunking. The extracted text is split into chunks, roughly the size of a few paragraphs each, sized to balance two competing goals: chunks small enough that each one is topically coherent (so a similarity match is meaningful), and large enough that each one carries enough surrounding context to be useful once retrieved.
3. Embedding. Each chunk is passed through an embedding model to produce its vector representation.
4. Indexing. The embeddings are stored in a structure that supports fast approximate nearest-neighbor search at query time.

You do not write code for any of these four steps yourself when using file_search — that automation is the entire value proposition versus the custom pipeline in Unit 10. What you can influence is the input quality (Lesson 7 covers preparing documents so this pipeline performs better) and, in some SDK versions, chunking parameters passed at attachment time.

Supported File Types and Practical Limits

file_search supports common document formats — PDF, DOCX, TXT, Markdown, and several others — but not every format is equally well-suited to automatic chunking. A clean, well-structured Markdown file with clear headings extracts and chunks far more reliably than a PDF that was originally a scanned image with no embedded text layer, because there is no text to extract from a pure image — that document would need OCR (as covered for PDFs generally in Unit 7) before it's usable here at all.

Every account also operates under storage and file-size limits that change over time as the platform evolves.

Note: Exact supported file formats, maximum file size per document, and maximum total storage per vector store are platform limits that change over time — check the current OpenAI documentation for these numbers before planning ingestion at scale.

Handling Upload Failures Gracefully

Because ingestion is asynchronous and can fail per-file (a corrupted PDF, an unsupported format, a file that exceeds a size limit), production ingestion code should never assume success:

def upload_documents(client, vector_store_id, file_paths):
    results = {"succeeded": [], "failed": []}

    for path in file_paths:
        try:
            with open(path, "rb") as f:
                outcome = client.vector_stores.files.upload_and_poll(
                    vector_store_id=vector_store_id,
                    file=f,
                )
            if outcome.status == "completed":
                results["succeeded"].append(path)
            else:
                results["failed"].append((path, outcome.status))
        except Exception as exc:
            results["failed"].append((path, str(exc)))

    return results

This function wraps each individual upload in a try/except block and separately tracks the outcome even when no exception is raised but the resulting status still isn't completed — a distinction that matters because ingestion failures often surface as a failed status rather than a Python exception. Structuring an ingestion pipeline this way means one bad file (a password-protected PDF, say) doesn't halt the entire batch, and you get a clear, actionable list of exactly which documents need attention afterward, rather than discovering gaps in your knowledge base only when a user asks about missing content.

You can test the failure-tracking logic itself, independent of any real API call, using a fake client:

class FakeOutcome:
    def __init__(self, status):
        self.status = status


class FakeFilesAPI:
    def __init__(self, statuses_by_path):
        self.statuses_by_path = statuses_by_path

    def upload_and_poll(self, vector_store_id, file):
        path = file.name
        status = self.statuses_by_path[path]
        if status == "raise":
            raise RuntimeError("simulated upload failure")
        return FakeOutcome(status)


class FakeVectorStoresAPI:
    def __init__(self, statuses_by_path):
        self.files = FakeFilesAPI(statuses_by_path)


class FakeClient:
    def __init__(self, statuses_by_path):
        self.vector_stores = FakeVectorStoresAPI(statuses_by_path)


def test_upload_documents_separates_success_and_failure():
    fake_client = FakeClient({
        "good.pdf": "completed",
        "bad.pdf": "failed",
        "broken.pdf": "raise",
    })

    results = upload_documents(
        fake_client, "vs_test", ["good.pdf", "bad.pdf", "broken.pdf"]
    )

    assert results["succeeded"] == ["good.pdf"]
    assert [path for path, _ in results["failed"]] == ["bad.pdf", "broken.pdf"]
    print("PASS: upload_documents separates success and failure correctly")


test_upload_documents_separates_success_and_failure()

This test builds fake stand-ins for the OpenAI client's nested vector_stores.files.upload_and_poll method, controlling exactly what each simulated file path returns, without making any real network call. It confirms that upload_documents correctly buckets a successful upload, a status-level failure, and an exception-raising failure into the right lists. This dependency-injection style — passing a fake object with the same shape as the real client — lets you verify your own ingestion logic in isolation, which is far faster and more reliable than testing against the live API every time you change the surrounding code.

Common Mistakes

Assuming files.create alone makes a document searchable, which it does not — a file only becomes part of a vector store's searchable index once explicitly attached via vector_stores.files.create (or a helper that does both steps), so an uploaded-but-unattached file will never be retrieved.

Not polling or checking status after a batch upload, leading to silent gaps in the knowledge base when some files fail to parse — always inspect file_counts.failed or per-file status after any ingestion run.

Re-uploading and re-attaching an entire document set on every deploy, wasting time and money on unchanged files — track which files are already successfully indexed (by storing file IDs and content hashes) and only upload what's new or changed.

Best Practices

Wrap every upload in error handling that records failures with enough detail to act on, including the file path and either the returned status or the exception message, so a failed ingestion run produces an actionable report rather than a silent partial success.

Prefer batch uploads for anything beyond a handful of files, since they reduce round trips and give you aggregate status in one call rather than requiring you to track many individual requests.

Keep a durable record, outside the vector store itself, of which source files have been uploaded and their resulting file IDs, so you can later delete, replace, or audit specific documents without having to reverse-engineer that mapping from the vector store's contents.