Horizontal vs. Vertical Scaling

When a service cannot handle its current load, there are two fundamentally different ways to give it more capacity. Vertical scaling means giving a single instance more resources — a bigger CPU allocation, more memory. Horizontal scaling means running more instances of the same service side by side, with a load balancer distributing requests across them. Vertical scaling has a hard ceiling (there is always a biggest machine available) and a single point of failure (that one bigger machine can still crash). Horizontal scaling has a much higher practical ceiling and, done correctly, no single instance whose failure takes down the whole service.

For an OpenAI SDK application, horizontal scaling is usually the more natural fit, for a reason specific to this kind of workload: most of the time a request spends "in progress" is spent waiting on a network response from the OpenAI API, not consuming your own CPU. Adding more CPU to one instance (vertical scaling) does very little for a workload that is mostly waiting on I/O — what actually helps is running more instances that can each be waiting on their own independent request at the same time.

The Precondition: Statelessness

Horizontal scaling only works cleanly if any instance can handle any request — which requires the application to be stateless: no instance-specific data that a later request depends on. If your application stores session data, job status, or any user-specific state only in that instance's memory, a load balancer routing a follow-up request to a different instance will find that state missing.

# Anti-pattern: state lives only in this process's memory.
_in_memory_sessions: dict[str, dict] = {}


def save_session(session_id: str, data: dict) -> None:
    _in_memory_sessions[session_id] = data  # invisible to every other replica


def load_session(session_id: str) -> dict | None:
    return _in_memory_sessions.get(session_id)  # only works if the SAME instance handles both calls

This pattern works perfectly on a single instance and breaks the moment you run two. The fix is to move any state that must persist between requests into a shared, external store — a database, Redis, or the durable job queue from Lesson 7 — that every replica can read from and write to identically.

def save_session(redis_client, session_id: str, data: dict) -> None:
    redis_client.set(f"session:{session_id}", data_to_json(data), ex=3600)


def load_session(redis_client, session_id: str) -> dict | None:
    raw = redis_client.get(f"session:{session_id}")
    return json_to_data(raw) if raw is not None else None


def data_to_json(data: dict) -> str:
    import json
    return json.dumps(data)


def json_to_data(raw: bytes | str) -> dict:
    import json
    return json.loads(raw)

Passing redis_client in as a parameter, rather than reaching for a module-level global, keeps this consistent with the dependency-injection pattern used throughout this course — and it also means these functions can be tested against an in-memory fake with the same .set / .get interface, without a real Redis instance running.

Shared Rate Limits Across Replicas

This is the scaling concern most specific to AI workloads, and it is worth connecting directly back to Unit 12, Lesson 3, which covered client-side rate limiting for a single process. A rate limiter that tracks "requests made in the last minute" using an in-process counter works correctly for exactly one running instance. The moment you scale horizontally to, say, five replicas, each replica has its own independent counter — each thinks it is respecting the limit, but the actual number of requests hitting the OpenAI API is the sum across all five, potentially five times what any single replica believes it is sending.

# Anti-pattern at scale: this limiter only sees its own process's traffic.
import time


class LocalRateLimiter:
    def __init__(self, max_per_minute: int) -> None:
        self._max = max_per_minute
        self._timestamps: list[float] = []

    def allow(self) -> bool:
        now = time.monotonic()
        self._timestamps = [t for t in self._timestamps if now - t < 60]
        if len(self._timestamps) >= self._max:
            return False
        self._timestamps.append(now)
        return True

LocalRateLimiter is exactly correct for a single process and exactly wrong for a fleet of replicas sharing one underlying API quota. The fix is to move the counting into a shared store all replicas consult, the same way session state moved into Redis above:

def allow_request(redis_client, key: str, max_per_minute: int) -> bool:
    """A simple fixed-window limiter shared across every replica via Redis.
    INCR is atomic, so concurrent replicas cannot both observe a stale count."""
    current = redis_client.incr(key)
    if current == 1:
        redis_client.expire(key, 60)
    return current <= max_per_minute

redis_client.incr(key) atomically increments a shared counter and returns the new value — "atomically" here is the operative word: even if two replicas call this at the exact same instant, Redis guarantees each increment is applied one at a time, so the count is never lost or double-counted the way it could be with a naive read-then-write in application code. Setting an expiry (redis_client.expire(key, 60)) only the first time the key is created (current == 1) turns this into a rolling one-minute window that resets automatically, without a separate cleanup process.

This is the concrete, practical version of "shared rate limits across replicas": whatever rate-limiting strategy Unit 12 introduced for a single process, its counting mechanism has to move to a data store every replica shares once you scale horizontally — otherwise your actual aggregate request rate to the OpenAI API silently scales with your replica count, which can turn a carefully tuned rate limit into one that is exceeded by a wide margin the moment you add a second instance.

What to Scale On

Traditional autoscaling often triggers on CPU utilization: add instances when average CPU crosses some threshold. For an AI workload dominated by waiting on an external API, CPU is frequently a poor signal — a replica can be handling many concurrent requests, each mostly idle waiting for OpenAI's response, while its CPU usage stays low the entire time. Scaling only on CPU under this pattern means you will not scale out even while genuinely overloaded, because the metric that would tell you so never moves.

More useful signals for this kind of workload include:

• In-flight request count or concurrency — how many requests each replica is currently handling, regardless of CPU usage.
• Queue depth — for the architecture from Lesson 7, how many jobs are waiting to be picked up by a worker is a direct measure of whether current worker capacity is keeping up with demand.
• P95/P99 request latency — a rising tail latency, even with average latency looking fine, often signals that some requests are queuing behind others faster than they can be served.

Common Mistakes

Storing per-user or per-session state only in an instance's local memory. This breaks the moment a load balancer routes a user's next request to a different instance, producing intermittent, confusing bugs that only appear in a multi-replica deployment and never show up when testing against a single local instance.

Rate-limiting only within a single process when running multiple replicas. As shown above, this silently multiplies your effective request rate to the OpenAI API by your replica count, which can produce a wave of 429 errors that seem to appear "out of nowhere" right after a scale-out event.

Autoscaling purely on CPU for an I/O-bound AI workload. This tends to under-scale during genuine overload (because CPU stays low while requests queue up waiting on external API responses) and can leave real users experiencing high latency while your monitoring shows healthy-looking CPU graphs.

Best Practices

Design every service intended to scale horizontally to be stateless, with all cross-request state — sessions, job status, rate-limit counters — held in a shared external store rather than a single process's memory.

Centralize rate limiting in a shared store (Redis or equivalent) as soon as you run more than one replica, so the limit reflects your true aggregate request volume to the OpenAI API rather than each replica's incomplete local view.

Scale on signals that reflect actual load for I/O-bound workloads — concurrency, queue depth, and tail latency — rather than defaulting to CPU utilization alone.