Why Some Work Cannot Happen Inside a Request

A web request has an implicit expectation: the client is waiting, and it will time out after some number of seconds (often 30, sometimes less behind a load balancer or API gateway). Many OpenAI SDK workloads do not fit inside that window — summarizing a hundred-page document, running a multi-step agent loop with several tool calls, transcribing a long audio file and then analyzing the transcript, or generating a large batch of images. If work like this runs directly inside an HTTP request handler, either the client's connection times out while the work is still in progress, or you are forced to hold a connection open far longer than is healthy for a web server designed to handle many short-lived requests concurrently.

A background worker is a separate process (or pool of processes) whose only job is to execute tasks handed to it, independent of any web request's lifetime. The web server's job becomes: accept the request, record that the work needs to happen, and immediately respond — typically with a job identifier the client can use to check on progress later. The actual OpenAI API calls happen in the worker process, on its own schedule, unconstrained by any request timeout.

This lesson is about the worker itself — what it is, why it is architected as a separate process, and how to build the simplest version of one. Lesson 7 builds on this by covering the queue that sits between the request handler and the worker in more depth: delivery guarantees, retries, and job status tracking as a first-class concern. It is worth distinguishing this material from Unit 21's batch-processing pipelines as well: Unit 21 covered patterns for processing many similar items together efficiently, such as OpenAI's dedicated Batch API for large offline jobs. This lesson is about general-purpose worker infrastructure for arbitrary long-running tasks submitted one at a time by user-facing requests — the underlying execution model, not the batch-specific cost and scheduling optimizations Unit 21 focused on.

The Simplest Possible Worker

Before reaching for a production task queue library, it helps to understand the core idea with the smallest implementation that demonstrates it: a loop that continuously checks for pending work and executes it.

import time
import uuid
import threading
from dataclasses import dataclass, field
from enum import Enum


class JobStatus(str, Enum):
    PENDING = "pending"
    RUNNING = "running"
    DONE = "done"
    FAILED = "failed"


@dataclass
class Job:
    id: str
    prompt: str
    status: JobStatus = JobStatus.PENDING
    result: str | None = None
    error: str | None = None


class JobStore:
    """In-memory job store, used here for teaching. Production systems use
    a durable store (a database or a queue's own storage) so jobs survive
    a process restart — see Lesson 7."""

    def __init__(self) -> None:
        self._jobs: dict[str, Job] = {}
        self._lock = threading.Lock()

    def create(self, prompt: str) -> Job:
        job = Job(id=str(uuid.uuid4()), prompt=prompt)
        with self._lock:
            self._jobs[job.id] = job
        return job

    def get(self, job_id: str) -> Job | None:
        with self._lock:
            return self._jobs.get(job_id)

    def next_pending(self) -> Job | None:
        with self._lock:
            for job in self._jobs.values():
                if job.status == JobStatus.PENDING:
                    job.status = JobStatus.RUNNING
                    return job
        return None

JobStore tracks jobs by an id and their current JobStatus. The _lock (a threading.Lock) matters because both the web server thread (creating jobs) and the worker thread (claiming and updating jobs) touch the same dictionary concurrently — without the lock, two threads could interleave their reads and writes and either lose a job or claim the same job twice. next_pending atomically finds a pending job and marks it RUNNING in the same locked section specifically to prevent two worker threads from both picking up the same job — a fresh pending job must be claimed exactly once.

The Worker Loop

def run_worker(client, store: JobStore, model: str, poll_interval: float = 1.0) -> None:
    """Continuously poll for pending jobs and execute them.
    In production this runs as its own process, separate from the web server."""
    while True:
        job = store.next_pending()
        if job is None:
            time.sleep(poll_interval)
            continue

        try:
            response = client.responses.create(model=model, input=job.prompt)
            job.result = response.output_text
            job.status = JobStatus.DONE
        except Exception as exc:
            job.error = str(exc)
            job.status = JobStatus.FAILED

This loop is intentionally simple, and every part of it maps to a real production concern. store.next_pending() returning None when there is no work means the worker sleeps for poll_interval seconds rather than spinning in a tight loop burning CPU for no reason — this is polling, the simplest way a worker can discover new work, at the cost of up to poll_interval seconds of latency between a job being created and a worker noticing it. Lesson 7 discusses queue systems that can push work to a worker instead of requiring it to poll, trading some implementation complexity for lower latency and less wasted CPU.

The try/except around the actual client.responses.create call is not incidental — it is the reason a worker architecture is more resilient than inline request handling in the first place. If this call raises (a timeout, a rate limit, a malformed prompt), the exception is caught, the job is marked FAILED with the error recorded, and — critically — the worker loop itself keeps running to process the next job. Without this try/except, one failing job would crash the entire worker process, taking down every other job queued behind it.

Running It End to End

class FakeResponse:
    def __init__(self, text: str) -> None:
        self.output_text = text


class FakeClient:
    class _Responses:
        def create(self, model: str, input: str) -> FakeResponse:
            return FakeResponse(f"Summary of: {input[:20]}")

    @property
    def responses(self) -> "FakeClient._Responses":
        return FakeClient._Responses()


def test_worker_processes_a_pending_job() -> None:
    store = JobStore()
    client = FakeClient()
    job = store.create("A very long document that needs summarizing.")

    # Run one iteration of the worker's core logic directly, rather than
    # starting the infinite loop, so the test terminates deterministically.
    pending = store.next_pending()
    assert pending is not None and pending.id == job.id

    response = client.responses.create(model="gpt-5.6-terra", input=pending.prompt)
    pending.result = response.output_text
    pending.status = JobStatus.DONE

    finished = store.get(job.id)
    assert finished is not None
    assert finished.status == JobStatus.DONE
    assert finished.result is not None and finished.result.startswith("Summary of:")
    print("PASS: worker processes a pending job end to end")


test_worker_processes_a_pending_job()

This test does not call run_worker directly, because run_worker contains an infinite while True loop that would never return. Instead, it exercises the same logic — claim a pending job, call the client, record the result — as a single, deterministic sequence of steps. This is a common and useful pattern when testing loop-based worker code: extract or replicate the loop's body into something testable, rather than trying to run the loop itself under test with an artificial exit condition.

Why the Worker Is a Separate Process

In a real deployment, run_worker does not share a process with your web server. It runs as its own container (built from the same image described in Lesson 3, but started with a different CMD — for example, CMD ["python", "worker.py"] instead of CMD ["uvicorn", ...]) or as a separately deployed service. This separation matters for a few concrete reasons: the web server can be scaled independently of the number of worker processes (Lesson 8 covers horizontal scaling in depth); a worker crash does not take down request handling for users who are not waiting on background jobs; and worker processes can be given different resource limits (more memory, longer allowed runtimes) appropriate to long-running AI tasks, without over-provisioning every web server replica the same way.

Note: Production systems rarely hand-roll a polling loop and an in-memory dict as shown here. Mature task-queue libraries — Celery, RQ, and Arq are common choices in the Python ecosystem — provide durable job storage (typically backed by Redis or a message broker), automatic retries, scheduled and periodic jobs, and worker pools out of the box. The teaching implementation in this lesson exists to make the underlying mechanism explicit before you adopt one of those libraries; the concepts (claim a job, execute it, record the outcome, keep the loop alive across individual failures) are the same either way.

Common Mistakes

Letting an unhandled exception in job processing crash the worker loop. Without a try/except around the actual task execution, one bad input can take down the worker entirely, silently stalling every job queued behind it until someone notices and restarts the process.

Running background work inside the request-handling thread or process instead of a genuinely separate one. This defeats the purpose — a slow job still blocks the web server's ability to handle other requests promptly, especially in a single-threaded or limited-worker-pool web server configuration.

Polling too aggressively. A poll_interval of zero or a few milliseconds turns the worker into a tight loop that consumes CPU and, if it is also checking a shared datastore, adds unnecessary load there for no meaningful latency benefit over a more reasonable interval.

Best Practices

Isolate the failure of one job from every other job with a try/except around task execution, so the worker loop's resilience does not depend on every individual task being bug-free.

Deploy workers as a separately scalable process from the web server, using the same container image but a different startup command, so the two can be resourced and scaled according to their very different workload characteristics.

Start with the simplest mechanism that works, and adopt a mature task-queue library once you need its guarantees — durable storage across restarts, automatic retries, and visibility into queue depth are exactly the concerns Lesson 7 covers next.