Home / AI Architecture & System Design / Agents & LLM Applications

Agents & LLM ApplicationsMidQuestion 6 of 9

How would you stream an LLM response to a web client?

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Functional requirements

Streaming looks like a front-end detail, but it touches authentication, cost, proxies, moderation and reliability. Pin down the product first.

You What's being streamed, and to which clients?

Interviewer A chat product on web and mobile. Responses stream token by token, sometimes with tool calls in between (searching docs).

You Do we moderate output before it's shown?

Interviewer Yes, there's an output moderation check.

You What happens if a user closes the tab mid-answer, or their connection drops?

Interviewer Today we keep generating, which wastes money. On mobile, connections drop a lot.

5–10 min Non-functional requirements and estimates

Back-of-envelope estimates:

• Concurrency: 5,000 new responses/s × ~10 s average stream = ~50,000 open streams at peak. Each holds a client connection and an upstream provider connection, so plan file descriptors, memory and load-balancer limits for ~100,000 connections.
• Bandwidth: ~50 tokens/s × ~4 bytes plus SSE framing (~40 bytes per event if every token is its own event) ≈ 2–3 KB/s per stream → ~150 MB/s total. Coalescing tokens into ~50 ms batches cuts the framing overhead a lot.
• Wasted generation: if 10% of users abandon mid-answer with ~300 tokens left, at 5,000 responses/s that's ~150,000 wasted output tokens/s without cancellation.

10–18 min High-level design

You The browser talks only to our backend over Server-Sent Events. The backend relays provider events as our own small event schema, keeps a buffer per message for resume, and cancels the provider request when the client goes away.

Components:

• Client: fetch with a ReadableStream (supports POST bodies and AbortController), a tolerant incremental Markdown renderer, and a stop button.
• Edge / reverse proxy: TLS, routing; buffering and compression disabled for the streaming route; idle timeouts longer than the longest silent period (tool calls).
• Streaming backend: authenticates, applies quotas, calls the provider's streaming API, maps events, coalesces tokens, writes each chunk to a short-lived message buffer, and forwards to the client.
• Message buffer: per-message log of chunks with sequence numbers in a fast store (e.g. Redis), kept a few minutes for resume.
• Moderation: checks windows of text before release (see deep dive).
• Usage log: tokens, cost, stop reason, latency per message.

Event schema (ours, not the provider's):

event: delta        data: {"seq": 41, "text": "The refund window is "}
event: tool_status  data: {"seq": 42, "tool": "search_docs", "state": "running"}
event: delta        data: {"seq": 43, "text": "30 days for annual plans."}
event: done         data: {"seq": 44, "stop_reason": "end_turn", "message_id": "m_123"}
event: error        data: {"seq": 45, "code": "upstream_unavailable", "retryable": true}

Our own schema means clients don't break when we change providers, and sequence numbers make resume possible.

18–32 min Deep dive

You I'd cover the buffering and timeout pitfalls, cancellation and resume, and how moderation fits with streaming.

Interviewer Users report answers arriving in big bursts after long pauses. Why?

Deep dive 1 · Buffering and timeouts

Almost always, something between the provider and the browser is buffering:

Also send a heartbeat (: ping) during long tool calls so intermediaries don't close the connection as idle.

Interviewer How does cancellation work end to end?

Deep dive 2 · Cancellation and resume

• User presses Stop: the client aborts the fetch. The backend detects the closed connection (or receives an explicit POST /messages/{id}/cancel for reliability) and closes the upstream stream, which stops generation and billing for further tokens. The partial message is saved with stop_reason: cancelled.
• Connection drops (not a stop): keep generating for a short grace period while buffering chunks. The client reconnects with Last-Event-ID: 43 (or ?after_seq=43), and the backend replays chunks 44+ from the buffer, then continues live.
• Distinguishing the two: an explicit cancel means stop; a silent disconnect means wait up to ~60 s for a reconnect, then cancel upstream.
• Server restarts: if the backend instance dies, the client reconnects to another instance; with chunks in a shared buffer it can serve what was generated, then mark the message incomplete and offer "continue".

Deep dive 3 · Moderation with streaming

• Hold and release by sentence: buffer until a sentence ends (or ~40 tokens), check it, then release. Adds a small delay per sentence but nothing unchecked is shown.
• Release and retract: for low-risk products, release immediately and send a retract event replacing text if a later check fails. Faster, but users may briefly see the text.
• Halt: on a violation, cancel upstream, emit a replacement message, and mark the turn so the partial text isn't fed back into the conversation.

Deep dive 4 · Rendering (if time allows)

• Incremental Markdown: partially received Markdown is often invalid (an open code fence). Use a renderer that tolerates incomplete input, or re-render the last block on each chunk.
• Coalescing: render at most every ~30–50 ms to avoid layout thrash on fast streams.
• Accessibility: announce completion to screen readers rather than every token.

32–37 min Trade-offs

Build first: the backend relay with our event schema, proxy settings and heartbeats, and cancellation. Then the resume buffer, then moderation windows.

37–40 min Wrap-up and likely follow-ups

Close with: "Clients stream from our backend over SSE using our own event schema with sequence numbers; the backend calls the provider with server-side keys, flushes per coalesced event, sends heartbeats during tool calls, and disables buffering on every hop. A per-message chunk buffer supports resume, explicit stops cancel upstream immediately, and silent disconnects get a short grace period. Moderation holds text per sentence before release."

Likely follow-ups:

• How do you load-balance long-lived connections? Least-connections balancing, connection limits per instance, and draining (stop accepting new streams, let existing ones finish) during deploys.
• What about serverless platforms? Check response-duration and streaming support limits; long agent tasks may need a separate long-running service.
• Mobile backgrounding? Rely on resume: when the app returns, reconnect with the last sequence number.

What interviewers listen for

• Keys server-side and a relay design with its own event format.
• Buffering, compression and idle-timeout pitfalls, with heartbeats.
• Cancellation that actually stops upstream generation, and a resume strategy.
• How moderation interacts with streaming.

Red flags

• Calling the provider directly from the browser with an API key.
• Ignoring client disconnects (paying for abandoned generations).
• "Just use WebSockets" without saying why.