A Running Process Is Not the Same as a Healthy One

A container orchestrator (or even plain Docker) can tell you whether a process is running. It cannot tell you, on its own, whether that process is actually able to do useful work. An OpenAI SDK service can be "running" while its event loop is deadlocked, while it has exhausted its connection pool, or while it started successfully but a required configuration value was silently defaulted to something wrong. Health checks exist to answer a more useful question than "is the process alive": is this instance able to correctly handle traffic right now?

This distinction is formalized in most container orchestration systems as two separate probes with different purposes: liveness and readiness.

Liveness vs. Readiness

A liveness check answers: "should this container be restarted?" It should be cheap, fast, and check only that the process's core event loop is responsive — not that every downstream dependency is working. If a liveness check fails repeatedly, the orchestrator's response is to kill and restart the container, on the theory that a stuck process is better replaced than left running.

A readiness check answers a different question: "should traffic be sent to this container right now?" A container can be alive (the process is fine) but not ready (a dependency it needs, like a shared cache or database, is temporarily unreachable). When a readiness check fails, the orchestrator stops routing new requests to that instance without killing it — giving it a chance to recover on its own once the dependency comes back.

Conflating the two is a common source of production incidents: a liveness check that verifies a downstream dependency will restart a perfectly healthy process every time that dependency has a hiccup, causing a self-inflicted outage exactly when things are already fragile.

Implementing Health Endpoints

For a FastAPI-based OpenAI SDK service, the conventional pattern is two lightweight HTTP endpoints:

import time
from fastapi import FastAPI, Response

app = FastAPI()
app.state.startup_complete = False
app.state.started_at = None


@app.on_event("startup")
def mark_startup_complete() -> None:
    # Configuration was already validated in create_app() (see Lesson 1).
    # Reaching this point means the process is ready to serve traffic.
    app.state.startup_complete = True
    app.state.started_at = time.monotonic()


@app.get("/healthz")
def liveness() -> dict:
    """Liveness: proves the event loop is responsive. No dependency checks."""
    return {"status": "alive"}


@app.get("/readyz")
def readiness(response: Response) -> dict:
    """Readiness: proves the app finished startup and can serve real requests."""
    if not app.state.startup_complete:
        response.status_code = 503
        return {"status": "not ready", "reason": "startup not complete"}

    has_api_key = bool(getattr(app.state, "config", None) and app.state.config.openai_api_key)
    if not has_api_key:
        response.status_code = 503
        return {"status": "not ready", "reason": "missing configuration"}

    return {"status": "ready"}

liveness() deliberately does almost nothing — it returns a fixed response the moment the request reaches it. If this endpoint responds at all, it proves the web server's request-handling loop is not deadlocked. It intentionally does not check the OpenAI API, a database, or any other dependency, because none of those failures mean this process is broken — they mean something else is, and killing this container would not fix that, only add a restart storm on top of it.

readiness() checks two things: that application startup fully completed (startup_complete), and that the configuration this instance needs is actually present. Neither check makes a network call. This is a deliberate and important design decision, not an oversight — the next section explains why.

Why Readiness Checks Should Not Call the OpenAI API

It is tempting to make the readiness check "more thorough" by having it actually call client.responses.create(...) to verify the OpenAI API is reachable and the key is valid. This is almost always the wrong choice, for three concrete reasons:

1. Cost. Orchestrators typically call the readiness endpoint every few seconds, for every running replica. If each check makes a real API call, you are paying for and rate-limiting against health checks, not real user traffic — and this cost scales with the number of replicas, which is exactly the situation Lesson 8 discusses under horizontal scaling.
2. Shared rate limits. A transient 429 from the OpenAI API — perhaps because another replica's real traffic briefly hit a rate limit — would mark this healthy replica as not ready, removing capacity from the pool at precisely the moment demand is high. This can cascade: fewer ready replicas means more load per replica, which increases the chance of more 429s, which marks more replicas not ready.
3. False negatives from transient network blips. A readiness check with a strict timeout calling an external API over the network is far more likely to fail for reasons that have nothing to do with whether your application can actually serve requests a moment later.

The general principle: readiness checks should verify your own process's internal state and locally available configuration, not the availability of a third-party API you do not control. If you genuinely need to detect sustained OpenAI API outages, that belongs in monitoring and alerting (Lesson 9), not in a per-replica readiness probe that controls traffic routing.

Startup Configuration: start-period and Grace Windows

Health checks need to account for the fact that an application takes some nonzero time to start — loading configuration, constructing clients, warming up any local caches. If the orchestrator starts checking readiness immediately and treats the first few failures as fatal, a perfectly normal, slightly slow startup looks like a broken deployment.

This is what the start-period in a Docker HEALTHCHECK (introduced in Lesson 3) and the equivalent initialDelaySeconds in other orchestrators are for — a grace window during which failed checks do not count against the container.

HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz', timeout=3)" || exit 1

Reading these parameters concretely: --start-period=15s gives the container 15 seconds after it starts before the first failing check counts at all — long enough for create_app() from Lesson 1 to run and configuration validation to complete. --interval=30s is how often the check runs afterward. --retries=3 means three consecutive failures are required before the container is marked unhealthy — a single slow response due to a brief garbage-collection pause should not trigger a restart. --timeout=5s bounds how long a single check attempt is allowed to take before it counts as a failure itself.

Note: Exact probe configuration syntax differs across platforms — Docker Compose, Kubernetes, and managed container platforms (AWS ECS, Google Cloud Run, Azure Container Apps) each express liveness, readiness, and startup grace periods slightly differently. The concepts in this lesson — cheap liveness, dependency-aware readiness, and a startup grace window — apply universally; only the configuration keys differ. Check your specific platform's current documentation for exact field names.

Testing Health Logic Without a Running Server

Because readiness() reads from app.state rather than performing I/O, its logic can be tested directly without starting an actual HTTP server:

class FakeState:
    def __init__(self, startup_complete: bool, has_key: bool) -> None:
        self.startup_complete = startup_complete
        self.config = type("Cfg", (), {"openai_api_key": "sk-fake" if has_key else ""})()


class FakeResponse:
    def __init__(self) -> None:
        self.status_code = 200


def readiness_logic(state: FakeState) -> tuple[dict, int]:
    response = FakeResponse()
    if not state.startup_complete:
        response.status_code = 503
        return {"status": "not ready", "reason": "startup not complete"}, response.status_code
    if not state.config.openai_api_key:
        response.status_code = 503
        return {"status": "not ready", "reason": "missing configuration"}, response.status_code
    return {"status": "ready"}, response.status_code


def test_not_ready_before_startup_completes() -> None:
    body, status = readiness_logic(FakeState(startup_complete=False, has_key=True))
    assert status == 503
    assert body["reason"] == "startup not complete"
    print("PASS: readiness reports not ready before startup completes")


def test_ready_once_startup_and_config_are_present() -> None:
    body, status = readiness_logic(FakeState(startup_complete=True, has_key=True))
    assert status == 200
    assert body["status"] == "ready"
    print("PASS: readiness reports ready once startup and config are complete")


test_not_ready_before_startup_completes()
test_ready_once_startup_and_config_are_present()

Extracting the decision logic into a plain function (readiness_logic) that takes a state object as a parameter — rather than testing the FastAPI route directly — follows the same dependency-injection pattern used throughout this course: the interesting behavior (what makes an instance "ready") is isolated from the web framework plumbing, so it can be tested in milliseconds with no server, no network, and no real configuration object.

Common Mistakes

Making the liveness check verify external dependencies. This causes an orchestrator to restart a perfectly healthy process because of a problem restarting cannot fix, often making an outage worse through repeated restart cycles.

Calling the real OpenAI API from a readiness probe. Beyond the direct cost of frequent API calls purely for health checking, this couples your traffic-routing decisions to the availability of a third-party service in a way that can cause cascading readiness failures under load, as described above.

Omitting a startup grace period. Without one, a container that takes a few extra seconds to initialize — entirely normal under load or during a cold start — can be marked unhealthy and killed before it ever gets a chance to serve a request.

Best Practices

Keep liveness checks trivially cheap and free of any dependency on external systems — they exist only to detect a genuinely stuck process.

Make readiness checks verify local state: configuration presence, internal startup completion, and any resources the process itself manages — not third-party API reachability.

Always configure a startup grace period sized generously enough for your slowest realistic startup path, and prefer requiring several consecutive failures (not a single blip) before an orchestrator takes action.