AI System DesignMidQuestion 2 of 12

How would you reduce the latency of an LLM feature that currently takes 8 seconds?

30-second answer

First measure the breakdown: retrieval, queueing, time-to-first-token, decode. Then: stream output; cut output length; use a smaller or faster model for some or all steps; parallelize retrieval and tool calls; trim the prompt and use prompt caching for static prefixes; cache frequent responses. If self-hosting, use continuous batching, quantization and speculative decoding.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

Architecture diagram

Sketch this on the whiteboard as you talk through the answer. See all AI architecture diagrams →

Scroll sideways to see the whole diagram.

Latency breakdown before and after optimization. Before, every step ran in sequence and nothing showed until 8.2 s. After a small rewrite model, fewer reranked chunks, shorter answers and streaming, text appears at 1.4 s and the answer finishes at 3.5 s.

0–5 min Clarify and frame

Ask: is 8s the time to first token or total? Is output streamed? What does the pipeline include (retrieval, tools, multiple LLM calls)? What's the target? Framing: "Measure before optimizing. I'll break latency into components, then apply the highest-leverage fixes for each."

5–15 min Core explanation: decompose

total ≈ network + queue + retrieval + (other LLM calls) + prefill (TTFT) + decode (output_tokens × per-token time) + post-processing

Instrument each span with tracing. Typical findings:

Decode dominates when answers are long.
Sequential chains (rewrite → retrieve → rerank → answer → verify) add up.
Long prompts inflate TTFT.
Rate-limit retries or queueing create tail latency.

Example breakdown: retrieval 400 ms, query-rewrite LLM call 900 ms, TTFT 1.2 s, decode 5.2 s (520 tokens at 100 tok/s), verification 300 ms.

15–25 min Deep dive: fixes by component

Component	Fixes
Perceived latency	Stream tokens; show progress for tool steps
Decode	Shorter outputs (format instructions, max tokens); smaller/faster model; speculative decoding if self-hosted
Prefill / TTFT	Trim prompt; fewer, reranked chunks; prompt caching for static prefix
Extra LLM calls	Use a small model for rewrite/classify; merge steps; skip when unnecessary (e.g. no rewrite on first turn)
Sequential steps	Parallelize independent retrieval and tool calls; start retrieval speculatively
Retrieval	ANN tuning, caching, colocate services
Tail latency	Timeouts, hedged requests, fallback models, capacity planning
Repeated queries	Response caching

Order by expected gain and effort; re-measure after each change.

25–32 min Trade-offs

Smaller models are faster but may reduce quality: check with evals.
Shorter answers may lose needed detail: offer "show more".
Parallel speculative work wastes compute when unused.
Caching risks staleness and personalization errors.

32–37 min Production example

Applying the breakdown above: switched rewrite to a small model (900 → 200 ms), streamed responses, limited answers to ~150 words with a details link (decode 5.2 s → 1.8 s), sent 4 reranked chunks instead of 12 (TTFT 1.2 → 0.6 s), and cached the system prompt. First visible text now appears at about 1.4 s instead of after the full 8 s, and the full answer finishes at about 3.5 s. Eval scores were unchanged.

37–40 min Likely follow-ups

What would you dashboard? p50/p95/p99 for TTFT, total, each span; output token distribution.
Self-hosted levers? Continuous batching, quantization, GQA models, speculative decoding, right-sized batch for latency SLOs.

What interviewers listen for

Measuring and decomposing first.
Knowing output tokens dominate and streaming improves perception.
Quality checks after optimizations.

Red flags

"Buy bigger GPUs" or "switch providers" as the first move.