Why "It Works" Is Not Enough Information

Once an OpenAI SDK application is deployed and handling real traffic, the question changes from "does it work" to "is it working right now, for everyone, and will I know the moment it stops." Monitoring is the practice of continuously collecting signals from a running system so that failures are detected — ideally before a user reports them — and so that when something does go wrong, there is enough information to diagnose it quickly rather than guessing.

This lesson focuses on what is specifically worth monitoring in an OpenAI SDK application, and on the logging discipline that makes an actual incident traceable after the fact. It builds on Lesson 1's introduction of structured logging in place of print().

What to Monitor

Error rate, broken down by error type. Not every failure is the same kind of problem: a spike in HTTP 429 responses from the OpenAI API means you are being rate-limited (connect this to Unit 12, Lesson 3's rate-limiting logic and Lesson 8's shared-rate-limit concern for horizontally scaled deployments); a spike in 500-level errors from the OpenAI API itself suggests an outage on OpenAI's side, which no amount of retrying on your end will fix; a spike in errors that are entirely your own application's exceptions (a KeyError, a validation failure) points at a bug in your code, not a dependency problem. Lumping all of these into one "error count" metric hides which of three very different responses is actually needed.

Latency, as percentiles, not just an average. An average latency of 800ms can hide the fact that 5% of your requests take 15 seconds — those users are having a genuinely bad experience that an average completely conceals. Tracking p50, p95, and p99 latency separately shows both the typical experience and the tail of it.

Token usage and cost. Because OpenAI API usage is billed by tokens, a sudden increase in token consumption — whether from a bug that causes unnecessarily long prompts, a runaway retry loop, or a genuine traffic increase — has a direct financial consequence. Tracking token usage as a first-class metric, not something you only discover by checking a billing dashboard days later, lets you catch a cost anomaly while it is still small.

Queue depth and worker throughput, for the architecture from Lessons 6 and 7. A queue depth that grows steadily over time, rather than staying roughly flat, means workers are falling behind incoming demand — a leading indicator of user-visible delay before any user has actually complained yet.

Structured Logging With Correlation IDs

A single user request in a production system often touches multiple log lines — a request received, an OpenAI API call made, a retry attempted, a response returned — potentially interleaved in the log stream with lines from other concurrent requests being handled by the same or other replicas. Without something tying those lines together, reconstructing what happened for one specific failing request is close to impossible once there is any real traffic volume.

The fix is a correlation ID (also called a request ID or trace ID): a unique identifier generated once per incoming request and included in every log line produced while handling it.

import logging
import uuid
import contextvars

request_id_var: contextvars.ContextVar[str] = contextvars.ContextVar(
    "request_id", default="-"
)


class RequestIdFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        record.request_id = request_id_var.get()
        return True


def configure_logging() -> None:
    handler = logging.StreamHandler()
    handler.addFilter(RequestIdFilter())
    handler.setFormatter(
        logging.Formatter(
            "%(asctime)s %(levelname)s [%(request_id)s] %(name)s %(message)s"
        )
    )
    root = logging.getLogger()
    root.addHandler(handler)
    root.setLevel(logging.INFO)


def new_request_id() -> str:
    request_id = str(uuid.uuid4())[:8]
    request_id_var.set(request_id)
    return request_id

contextvars.ContextVar is what makes this work correctly under concurrency: it holds a value that is specific to the current logical task (an async request handler, in most modern Python web frameworks) rather than a single global variable shared across every concurrent request, which would let one request's ID bleed into another's log lines under load. new_request_id() is called once, at the very start of handling each incoming request — typically in middleware — and every subsequent logger.info(...) call anywhere in that request's code path automatically picks up the same id through RequestIdFilter, without every function needing to explicitly pass the id down through every layer of the call stack.

import time

logger = logging.getLogger("myapp.requests")


def handle_summarize_request(client, model: str, document_text: str) -> str:
    start = time.monotonic()
    logger.info("summarize started doc_length=%d", len(document_text))
    try:
        response = client.responses.create(
            model=model, input=f"Summarize:\n{document_text}"
        )
    except Exception:
        elapsed_ms = (time.monotonic() - start) * 1000
        logger.exception("summarize failed after %.0fms", elapsed_ms)
        raise

    elapsed_ms = (time.monotonic() - start) * 1000
    logger.info("summarize completed elapsed_ms=%.0f", elapsed_ms)
    return response.output_text

Every log line in handle_summarize_request will carry the same request_id once configure_logging and new_request_id are wired into the request pipeline, which means that grepping (or, more realistically, querying a log-aggregation platform) for a single request id retrieves the complete story of that one request — start, any failure, and completion — even in a system handling thousands of concurrent requests across multiple replicas.

Classifying and Handling OpenAI API Errors

Not all exceptions from the OpenAI SDK should be treated identically for monitoring purposes. A useful pattern is to classify errors into categories your monitoring and alerting can distinguish between:

from enum import Enum


class ErrorCategory(str, Enum):
    RATE_LIMITED = "rate_limited"
    UPSTREAM_ERROR = "upstream_error"      # OpenAI-side 5xx
    CLIENT_ERROR = "client_error"          # our request was malformed (4xx, not 429)
    UNKNOWN = "unknown"


def classify_openai_error(status_code: int | None) -> ErrorCategory:
    if status_code == 429:
        return ErrorCategory.RATE_LIMITED
    if status_code is not None and 500 <= status_code < 600:
        return ErrorCategory.UPSTREAM_ERROR
    if status_code is not None and 400 <= status_code < 500:
        return ErrorCategory.CLIENT_ERROR
    return ErrorCategory.UNKNOWN


def test_classifies_rate_limit_and_upstream_errors() -> None:
    assert classify_openai_error(429) == ErrorCategory.RATE_LIMITED
    assert classify_openai_error(503) == ErrorCategory.UPSTREAM_ERROR
    assert classify_openai_error(400) == ErrorCategory.CLIENT_ERROR
    assert classify_openai_error(None) == ErrorCategory.UNKNOWN
    print("PASS: error classification maps status codes to the right category")


test_classifies_rate_limit_and_upstream_errors()

Logging (and alerting on) the category, not just "an error occurred," is what lets a human glance at a dashboard and immediately know what kind of response is warranted: a burst of RATE_LIMITED errors points back at the rate-limiting and scaling discussion in Unit 12 and Lesson 8 of this unit; a burst of UPSTREAM_ERROR means checking OpenAI's status page, not your own code; and any CLIENT_ERROR volume above baseline suggests a bug in how your application is constructing requests, worth investigating directly.

Alerting Without Alert Fatigue

An alert that fires too often for conditions that do not actually require action trains whoever receives it to ignore it — which means the one time it fires for something that genuinely matters, it gets ignored along with all the noise. Effective alerting thresholds are set based on what actually requires a human response, not simply "any error at all":

• Alert on a sustained elevated error rate over a window (for example, error rate above 5% for five consecutive minutes), not on any single error, which is expected to happen occasionally even in a healthy system.
• Alert on p99 latency crossing a threshold that reflects real user impact, not on p50 latency, which is far more sensitive to normal variation.
• Alert on queue depth that is both high and growing, not merely nonzero — a queue with a few items that are being processed steadily is normal; one that keeps growing is not.

Common Mistakes

Treating every exception identically in logs and alerts. Without classification, a dashboard showing "47 errors in the last hour" gives no indication of whether that means "OpenAI had a brief outage" or "we shipped a bug that breaks every request" — two situations with completely different responses.

Logging without a correlation id. When an incident report says "a user got an error around 2:15 PM," reconstructing exactly what happened from an unstructured log stream with no way to isolate that one request's lines is far slower than it needs to be, often too slow to matter by the time you find it.

Setting alert thresholds so sensitive that they fire on normal variation. This produces alert fatigue, where genuine, actionable alerts get the same "probably nothing" reaction as the noise around them, delaying real incident response.

Best Practices

Attach a correlation id to every log line for a given request, generated once per request and propagated automatically through a context variable, so any single request's full history can be reconstructed after the fact.

Classify failures by cause, distinguishing rate limiting, upstream provider errors, and your own application's bugs, so monitoring dashboards and alerts point directly at the appropriate response instead of an undifferentiated error count.

Track token usage and cost as an operational metric, not only a billing-dashboard afterthought, so a cost anomaly is caught while it is still small rather than discovered at the end of a billing cycle.