How would you stream an LLM response to a web client?

30-second answer

Call the provider's streaming endpoint (server-sent events) from the backend, never exposing API keys to the browser. Forward text deltas to the client via SSE or WebSockets, flush promptly, handle client disconnects by cancelling the upstream request, buffer tool-call arguments until complete, and record usage and errors from the final events.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

Architecture diagram

Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →

Scroll sideways to see the whole diagram.

Streaming LLM responses to the browser. The browser never holds the provider key. The backend relays provider events as its own SSE events, the proxy must not buffer, and a client disconnect cancels the upstream request.

0–5 min Clarify and frame

Ask about the stack (framework, hosting, proxies), whether tools or moderation are involved, and mobile clients. Framing: "The backend streams from the provider and relays events to the browser over SSE, keeping keys server-side, handling cancellation, errors and buffering pitfalls."

5–15 min Core explanation

Browser ──POST /chat (fetch, ReadableStream)──▶ Backend ──streaming API (SSE)──▶ LLM provider
        ◀──────── SSE: delta / tool_status / done / error ◀── relay + transform ◀── events

Backend responsibilities:

Authenticate the user, enforce quotas and rate limits.
Call the provider's streaming endpoint with your server-side API key.
Forward text deltas immediately; flush each event; disable proxy buffering.
Translate provider events into your own small event schema (delta, tool_status, done, error) so clients don't depend on vendor formats.
Buffer tool-call argument fragments until complete before parsing JSON.
Record usage and stop reason from the final events.
Cancel the upstream request when the client disconnects, to save cost.

15–25 min Deep dive: code

@app.post("/chat")
async def chat(req: ChatRequest, request: Request, user=Depends(auth)):
    check_quota(user)

    async def events():
        try:
            async with client.messages.stream(model=MODEL, max_tokens=1024,
                                              system=SYSTEM, messages=req.messages) as stream:
                async for text in stream.text_stream:
                    if await request.is_disconnected():
                        break                      # exiting the context closes upstream
                    yield sse({"type": "delta", "text": text})
                final = await stream.get_final_message()
                log_usage(user, final.usage)
                yield sse({"type": "done", "stop_reason": final.stop_reason})
        except ProviderError as e:
            yield sse({"type": "error", "message": "Something went wrong. Please retry."})

    return StreamingResponse(events(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

Frontend:

const res = await fetch("/chat", { method: "POST", body: JSON.stringify(payload), signal: ctrl.signal });
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
// parse "data: {...}\n\n" frames, append deltas, render Markdown incrementally
// Stop button → ctrl.abort()

EventSource is simpler but only supports GET; fetch streaming supports POST bodies and abort.

25–32 min Trade-offs and pitfalls

Buffering: reverse proxies, CDNs and compression middleware can hold responses until complete. Disable for the streaming route.
Timeouts: load balancers and serverless platforms may cap response duration; check limits for long generations.
SSE vs. WebSockets: SSE is one-way and proxy-friendly; WebSockets suit bidirectional, interruptible sessions (voice, collaborative editing).
Moderation: buffer to sentence boundaries before release, or release immediately and retract on violation.
Markdown rendering: partial Markdown (unclosed code fences) needs a tolerant renderer.
Resumability: persist generated text with a message ID so a reconnecting client can fetch the remainder.

32–37 min Production example

Users saw responses appear in big bursts after long pauses. The cause was an nginx proxy buffering responses plus gzip middleware. Disabling buffering (X-Accel-Buffering: no) and compression for the streaming route fixed it. A second fix cancelled upstream generation on client disconnect, which reduced token spend from abandoned requests.

37–40 min Likely follow-ups

How do you stream tool progress? Emit tool_status events ("Searching orders…") while tools run.
How do you test it? Replay recorded provider streams through the backend; end-to-end tests asserting incremental delivery.

What interviewers listen for

Server-side keys and a relay design with your own event schema.
Cancellation, buffering and timeout pitfalls.

Red flags

Calling the provider directly from the browser with a secret key.