Why Streaming Changes the User's Experience

Lesson 2's /chat endpoint waits for the model's entire response to finish generating before returning anything at all — for a short answer this is barely noticeable, but for a longer, more detailed answer grounded in retrieved document context, a user can be left staring at nothing for several seconds before any text appears. Streaming changes this by sending back each piece of the response as it's generated, so a user sees text appearing progressively rather than waiting for the complete answer — the total time to finish generating is the same either way, but the perceived responsiveness is considerably better, since the user has something to read within a fraction of a second rather than waiting for the entire answer to be ready.

Streaming a Response From the Model

The Responses API supports streaming directly, returning a sequence of events as the response is generated rather than a single completed object.

async def stream_model_response(async_client, messages: list):
    async with async_client.responses.stream(
        model="gpt-5.6-terra",
        input=messages,
    ) as stream:
        async for event in stream:
            if event.type == "response.output_text.delta":
                yield event.delta

Note: The exact streaming interface, the specific event type names, and which event carries incremental text can vary by SDK version. Confirm the current streaming API and event structure against your installed SDK version's documentation before relying on a specific event type name in production code.

Iterating over stream yields a sequence of events describing what's happening as the response is generated — checking specifically for response.output_text.delta events and yielding just the incremental text (event.delta) is what turns the full event stream, which includes other event types not relevant to display, into exactly the piece-by-piece text a frontend needs to show progressively.

Exposing a Streaming Endpoint With FastAPI

FastAPI's StreamingResponse takes an async generator and streams its output to the client as it's produced, which is exactly the shape stream_model_response() above already provides.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

async def sse_formatted_stream(async_client, messages: list):
    async for text_chunk in stream_model_response(async_client, messages):
        yield f"data: {text_chunk}\n\n"
    yield "data: [DONE]\n\n"

@app.post("/chat/stream")
async def chat_stream(request: ChatRequest):
    history = conversation_store.setdefault(request.conversation_id, [])
    relevant_chunks = await retrieve_relevant_chunks(request.message)
    context_block = "\n\n".join(relevant_chunks) if relevant_chunks else "No relevant document content found."

    messages = [
        {"role": "system", "content": f"Answer using only this context:\n{context_block}"},
        *history,
        {"role": "user", "content": request.message},
    ]

    return StreamingResponse(
        sse_formatted_stream(async_client, messages),
        media_type="text/event-stream",
    )

sse_formatted_stream() wraps each text chunk in the Server-Sent Events (SSE) format — a data: prefix followed by two newlines — which is a simple, widely supported convention for streaming text to a browser over a regular HTTP connection, with a final [DONE] marker signaling that no more chunks are coming. Reusing Lesson 2's retrieve_relevant_chunks() and conversation-history logic unchanged here reflects that streaming only changes how the final response is delivered, not how retrieval or conversation state work — everything from Lesson 2 up through generating the messages list carries over exactly as written.

A Simple Frontend Consuming the Stream

A minimal browser-side implementation reads the streamed response incrementally and appends each chunk to the page as it arrives, rather than waiting for the entire response body.

async function sendMessage(conversationId, message) {
  const response = await fetch("/chat/stream", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ conversation_id: conversationId, message: message }),
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  const replyElement = document.getElementById("reply");
  replyElement.textContent = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const chunkText = decoder.decode(value);
    for (const line of chunkText.split("\n\n")) {
      if (!line.startsWith("data: ")) continue;
      const data = line.slice(6);
      if (data === "[DONE]") return;
      replyElement.textContent += data;
    }
  }
}

response.body.getReader() gives direct access to the HTTP response body as it arrives, rather than waiting for fetch() to resolve with a complete body — each call to reader.read() returns whatever has arrived since the last call, which is decoded and parsed for data: lines matching the SSE format the backend produces. Appending each piece of text directly to replyElement.textContent as it's decoded is what produces the progressive, word-by-word (or chunk-by-chunk) appearance a streaming interface is meant to provide.

Handling Tool Calls Within a Streamed Response

Lesson 1's design includes a tool alongside retrieval, and a streamed response needs to account for a tool call occurring partway through generation rather than assuming every response is pure incremental text from start to finish.

async def stream_with_tool_awareness(async_client, messages: list, tools: list):
    async with async_client.responses.stream(
        model="gpt-5.6-terra",
        input=messages,
        tools=tools,
    ) as stream:
        async for event in stream:
            if event.type == "response.output_text.delta":
                yield {"type": "text", "content": event.delta}
            elif event.type == "response.function_call_arguments.delta":
                yield {"type": "tool_call_in_progress", "content": None}

Note: The exact event types emitted during a streamed response that includes a tool call, and how a completed tool call's result re-enters the stream, can vary by SDK version. Confirm the current event sequence for tool-calling streams against your installed SDK version's documentation before building production logic around a specific event type.

Distinguishing a response.function_call_arguments.delta event from a text delta matters for the frontend's experience: rather than showing nothing (or a raw, confusing partial function call) while a tool is being invoked, the frontend can show an indicator — "checking loaded documents..." — that something is happening, closing the same kind of perceived-latency gap streaming addresses for text generation, applied to the tool-calling step instead.

Common Mistakes

Buffering the entire response server-side before sending anything to the client, defeating the purpose of streaming even though the endpoint is nominally using a streaming API.

Assuming every streamed response consists only of text delta events, missing tool-call-related events entirely and producing a confusing or broken experience when the model happens to call the tool mid-response.

Not handling a dropped or interrupted connection on the frontend, leaving a user's interface stuck mid-response with no indication that the stream ended unexpectedly rather than completing normally.

Parsing the SSE stream incorrectly — such as assuming each read() call returns exactly one complete data: line — when a single chunk from the underlying connection can contain a partial line, multiple lines, or split awkwardly across chunk boundaries.

Best Practices

Stream text to the frontend incrementally as it's generated, rather than waiting for the full response, to substantially improve perceived responsiveness for longer answers.

Reuse the same retrieval and conversation-state logic between streaming and non-streaming endpoints, changing only how the final response is delivered.

Give the frontend a visible indicator for non-text events, such as an in-progress tool call, rather than leaving the interface silent while something is happening behind the scenes.

Handle stream parsing defensively, accounting for a chunk boundary that doesn't align cleanly with a complete SSE data: line.