What Changes Between Local Development and a Real Deployment

Every lesson in this capstone so far has run against a local development setup — an API key available as an environment variable, in-memory storage that resets every time the process restarts, no real users other than whoever is testing it. Moving toward an actual deployment means revisiting several of these choices deliberately, and this lesson works through what needs to change, organized as a checklist that draws together practices from across this course rather than introducing substantial new material of its own.

Configuration and Secrets

The API key should never be hardcoded into source code at any point, including during local development, since code has a way of ending up somewhere it wasn't meant to (a public repository, a shared screen) far more easily than an environment variable does.

import os
from openai import AsyncOpenAI

# Never do this:
# async_client = AsyncOpenAI(api_key="sk-...")

# Instead, rely on the OPENAI_API_KEY environment variable (the SDK reads it
# automatically), or read a deployment-specific secret explicitly:
async_client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])

For an actual deployment, this environment variable should come from the hosting platform's own secrets management (rather than a checked-in .env file), and the same principle extends to any other credential the application depends on — a database connection string, for instance, once Lesson 1's in-memory storage is replaced with something persistent.

Replacing In-Memory Storage

Lesson 1 deliberately scoped this capstone to use in-memory storage for document chunks and conversation history, explicitly noting that a real deployment would need something persistent — this is the point where that substitution actually needs to happen.

# In-memory (this capstone's scope): lost on every restart, not shared across
# multiple server instances.
document_store: dict = {}

# A real deployment needs a database (or a dedicated vector database for the
# embeddings specifically) so that data survives restarts and is visible
# consistently across every server instance handling requests.

This substitution matters for two distinct reasons, not just one: data surviving a restart is the more obvious concern, but a deployed application typically runs more than one server instance for reliability and load handling, and an in-memory dictionary in one instance's process is invisible to every other instance — a document uploaded through one instance would simply not be found by a chat request handled by a different instance.

Applying Unit 12's Production-Readiness Practices

Everything Unit 12 covered — error handling, retries, timeouts, rate limiting — applies directly to this capstone's actual deployment, and skipping any of it here reproduces exactly the gaps that unit worked through in the abstract.

from openai import AsyncOpenAI

async_client = AsyncOpenAI(
    max_retries=3,     # Unit 12, Lesson 2
    timeout=30.0,       # Unit 12, Lesson 2
)

Beyond client configuration, the application-level rate limiter from Unit 12, Lesson 3 becomes genuinely necessary once real users are making requests concurrently rather than a single developer testing one request at a time — without it, a burst of simultaneous chat requests across many users could exceed the account's actual rate limit in a way that never surfaced during local development.

Applying Unit 12's Moderation Guidance

A document assistant accepting arbitrary user-uploaded content and arbitrary user questions is exactly the kind of application Unit 12, Lesson 7's moderation guidance was written for — checking both what a user submits and what the assistant generates before it reaches another user.

async def is_content_safe(async_client, text: str) -> bool:
    result = (await async_client.moderations.create(input=text)).results[0]
    return not result.flagged

@app.post("/chat")
async def chat_with_moderation(request: ChatRequest):
    if not await is_content_safe(async_client, request.message):
        raise HTTPException(status_code=400, detail="This message doesn't meet content guidelines.")
    # ... proceed with retrieval and response generation as in Lesson 2

This check is easy to skip during local development, where the only inputs being tested are the developer's own — it becomes necessary the moment the application is reachable by anyone else, exactly the transition this lesson is about.

A Pre-Launch Checklist

Pulling every consideration above (and a few additional ones specific to launching) together into one concrete list to work through before an actual deployment:

1. Secrets: API keys and any database credentials come from the hosting platform's secrets management, never hardcoded or committed to source control.
2. Persistence: document storage and conversation history use a real database rather than the in-memory stores this capstone used for clarity during development.
3. Client configuration: max_retries and timeout are set explicitly (Unit 12, Lesson 2) rather than relying on undocumented defaults.
4. Rate limiting: an application-level rate limiter (Unit 12, Lesson 3) is in place if concurrent usage could plausibly approach the account's actual rate limit.
5. Error handling: every endpoint catches upstream API errors and translates them into meaningful HTTP responses (Lesson 2), rather than allowing an unhandled exception to reach the client as a generic failure.
6. Moderation: user input and, where practical, model output are checked against Unit 12, Lesson 7's moderation guidance before an application is reachable by real users.
7. Logging: enough is logged to diagnose a production issue after the fact — which requests failed and why — without over-retaining sensitive content, following Unit 12, Lesson 7's data-handling caution.
8. Cost awareness: at minimum, max_output_tokens is capped to a sensible bound, and the system prompt's shared, static portions are structured to take advantage of prompt caching (Unit 12, Lesson 4) where the request volume would make that meaningful.

A Brief Cost Checklist Beyond What's Already Covered

Unit 12, Lesson 4 covered prompt caching and cost optimization at length; two points specific to this capstone's shape are worth calling out directly before launch. First, retrieval calls an embeddings model on every single chat message to embed the incoming question — this is a small, cheap call individually, but it's worth confirming it isn't being made redundantly (embedding the same repeated question twice within one request, for instance). Second, the system instructions built in Lesson 2 include the retrieved document context as part of the prompt on every turn — following Unit 12, Lesson 4's caching guidance, structuring a prompt with any genuinely static portions (like the base instruction text, separate from the per-turn retrieved context) first would let that static portion benefit from caching, even though the retrieved context itself necessarily changes from turn to turn and can't be cached the same way.

Common Mistakes

Deploying with in-memory storage unchanged, losing all data on every restart and producing inconsistent behavior across multiple server instances handling different requests.

Skipping application-level rate limiting because it was never needed during single-developer local testing, only to discover the account's actual rate limit under real concurrent usage.

Adding moderation and safety checks as an afterthought post-launch, rather than as part of the pre-launch checklist before an application is reachable by anyone outside development.

Treating Unit 12's production-readiness practices as optional polish rather than integral to an actual deployment, when skipping them reproduces the exact gaps that unit was written to close.

Best Practices

Work through a concrete pre-launch checklist before deploying, rather than assuming a locally-working application is automatically ready for real users.

Replace in-memory storage with real persistence before any deployment involving more than one server instance or requiring data to survive a restart.

Apply Unit 12's retry, timeout, rate-limiting, and moderation practices as a baseline for any deployed application, not as optional additions reserved for large-scale systems.

Revisit cost structure specifically for this application's shape, checking for redundant embedding calls and structuring prompts to take advantage of caching where request volume justifies it.