Home / AI Architecture & System Design / AI System Design

AI System DesignStaffQuestion 13 of 14

Design an LLM inference service that serves open-weight models on your own GPUs to internal teams.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Functional requirements

"Inference service" can mean a thin proxy to an API provider or a GPU fleet you run yourself. Establish which before anything else.

You Are we serving models ourselves on our own GPUs, or routing to external APIs?

Interviewer Ourselves. Open-weight models on our own H100 cluster, for internal product teams.

You Which models, and how many?

Interviewer Start with an 8B model for high-volume, simple tasks and a 70B model for quality-sensitive ones. More models will come, including fine-tuned variants.

You What do clients need? Chat completions with streaming, tool calls, structured output, embeddings?

Interviewer Chat completions with streaming and JSON-schema output, through an OpenAI-compatible API so teams can switch easily.

You Is there offline batch work too, like nightly summarization?

Interviewer Yes, a fair amount. It can be slow but should be cheap.

Out of scope: training, external-provider fallback (mention as an extension), multimodal input.

5–10 min Non-functional requirements and estimates

You What latency do interactive clients need?

Interviewer First token within about a second at p95 for typical prompts, and streaming at least as fast as people read.

You I'll set TTFT ≤ 800 ms p95 for prompts up to 4k tokens and time per output token ≤ 40 ms p95, which is 25 tokens/s, comfortably faster than reading speed. The time-per-token target caps how big decode batches can get, so it directly sets cost.

Back-of-envelope estimates (peak interactive traffic, 2,000 input and 300 output tokens on average):

• Split: 70% of requests to the 8B model (210 req/s), 30% to the 70B (90 req/s).
• 70B decode: 90 × 300 = 27,000 output tokens/s. From roofline analysis, an 8 × H100 node (TP = 8) sustains roughly 6,000 tokens/s within a 40 ms per-token budget in bf16. That's ~4.5 nodes of decode.
• 70B prefill: 90 × 2,000 = 180k tokens/s × 2 × 70e9 FLOPs ≈ 2.5e16 FLOP/s. At ~50% utilization a node delivers ~4e15, so ~6 nodes of compute without caching; with ~50% prefix-cache hits (shared system prompts), ~3.
• 8B: 63k output tokens/s at ~4,000 tokens/s per GPU ≈ 16 GPUs; prefill 420k tokens/s × 1.6e10 ≈ 6.7e15 FLOP/s ≈ 7 GPUs with caching.
• Fleet: ~8 nodes for the 70B and ~3 nodes for the 8B, plus ~30% headroom for failures and spikes: ~14 nodes (112 GPUs) at peak. Off-peak, much of this is free for batch work.
• Weights: 70B in bf16 is 140 GB. Loading from object storage at ~2 GB/s takes over a minute; from local NVMe at ~5 GB/s, ~30 s. Cold starts are slow, which shapes autoscaling.

10–18 min High-level design

You Three layers: a stateless API layer that authenticates, meters and validates; a router that picks a replica for each request using cache affinity and load; and model pools of inference engines doing continuous batching. A control plane manages model versions, autoscaling and rollouts.

Components:

• API gateway: authentication, OpenAI-compatible schema, per-team token buckets, request validation (max context, max output), SSE streaming to clients.
• Router: maps the model name to a pool and version, then picks a replica by prefix-cache affinity and current load (outstanding tokens, KV utilization). Interactive and batch requests have separate queues.
• Model pools: replicas of an inference engine (continuous batching, paged KV cache, prefix caching, CUDA graphs). 70B replicas use TP = 8 on one node; 8B replicas use one GPU each.
• Batch service: accepts batch jobs, stores inputs in object storage, and feeds requests to the router at low priority when capacity is free.
• Model registry and weight store: versioned weights (safetensors) with checksums, quantized variants and engine configs; nodes cache weights on local NVMe.
• Autoscaler: scales pools on queue depth, KV-cache utilization and SLO burn rate, with pre-warmed spares.
• Metering and observability: per-request tokens, TTFT, time per token, queue time, cache hits, by team, model and version.

One interactive request:

1. The gateway authenticates the team, checks its token bucket (estimated input tokens plus max output), and validates the request.
2. The router hashes the first blocks of the prompt (system prompt and shared context) and prefers the replica that's likely to hold them in its prefix cache, unless that replica is overloaded.
3. The replica admits the request when KV blocks are available; otherwise it waits in the replica queue (bounded) or the router tries another replica.
4. Prefill runs in chunks interleaved with ongoing decodes; tokens stream back through the gateway.
5. On completion, usage is recorded and the token bucket is reconciled with actual tokens.

API sketch:

POST /v1/chat/completions   { model: "chat-large", messages, stream, response_format, max_tokens, priority }
     → SSE: delta … | usage | done
POST /v1/batches            { model, input_file, completion_window: "12h" } → { batch_id }
GET  /v1/models             → names, versions, context limits

18–32 min Deep dive

You I'd go deeper on scheduling and routing, then capacity and autoscaling, then rollouts and failure handling.

Interviewer Start with scheduling. How do you hit the latency targets while keeping GPUs busy?

Deep dive 1 · Scheduling within a replica

• Continuous batching: sequences join and leave the batch every iteration. The batch size is bounded by two things: free KV blocks, and the time-per-token SLO. Measure step time as a function of batch size and context length per model and set a maximum batch that keeps p95 under 40 ms.
• Chunked prefill: a 30k-token prompt would stall every decode in the batch for hundreds of milliseconds. Prefill in chunks of ~2k tokens per iteration, so decodes keep their cadence and the long prompt still progresses.
• Prefix caching: system prompts and few-shot examples are shared by many requests from the same team. Cache KV blocks by content hash; a hit skips that part of prefill, which reduces TTFT and compute. Keep cached blocks in an LRU that yields to active sequences.
• Priorities: batch requests only take slots when interactive queues are empty and are preempted first (their KV blocks are freed; they're recomputed later). That keeps utilization high off-peak without affecting interactive latency.
• Admission control: when queue wait would break the TTFT target, reject early with 429 and Retry-After rather than accepting requests that will time out.

Interviewer How does the router choose a replica?

Deep dive 2 · Routing

• Cache affinity: consistent hashing on a hash of the first N prompt blocks maps requests with the same prefix to the same replica, raising prefix-cache hit rates.
• Bounded load: pure hashing creates hot spots (one team's popular prompt). Use consistent hashing with bounded loads: if the preferred replica is above, say, 1.25× the pool's average load, go to the next one.
• Load signal: outstanding tokens (prompt tokens left to prefill plus expected decode tokens) is better than request count, because requests vary 100× in cost. Replicas report KV utilization and queue depth every few hundred milliseconds.
• Fairness: per-team token buckets at the gateway, and weighted fair queuing in the router so a burst from one team doesn't monopolize a pool.

Deep dive 3 · Capacity, autoscaling and disaggregation

• Signals: scale up on interactive queue time and KV utilization (leading indicators), not GPU utilization (always high with batching).
• Cold starts: a 70B replica takes ~1–2 minutes (node allocation, weight load, CUDA-graph capture, warm-up). Keep a small pool of warm spares, scale ahead of known daily peaks, and use batch traffic as the buffer: preempt it when interactive load rises.
• Weight distribution: pre-stage popular model weights on node NVMe; stream weights from peers rather than object storage when many nodes scale at once.
• Disaggregation (when prompts are long): for the 70B pool, separate prefill and decode replicas. Prefill nodes run large compute-bound batches; decode nodes run memory-bound batches tuned for time per token; KV caches transfer over the fast network. It decouples TTFT from time per token and lets each side scale on its own signal, at the cost of KV transfer bandwidth and complexity. Start without it; adopt it when chunked prefill can't hold both SLOs.
• Quantization: fp8 weights for the 70B halve weight reads per step and raise per-node throughput, if evaluations show no regression on team-specific tasks.

Deep dive 4 · Rollouts and failures

• New model or engine version: deploy to a canary replica set; mirror a sample of traffic for golden-output comparison (judge-based quality checks, JSON validity, refusal rate) and compare latency; then shift 5% → 25% → 100% with automatic rollback on SLO or quality regressions. Clients that pin versions are unaffected.
• Replica failure mid-stream: a stream that has already sent tokens can't be transparently retried without duplicating text. Return an error event with the partial usage, and let clients retry; for non-streaming requests the gateway retries on another replica.
• Node health: GPU errors (ECC, Xid), NCCL timeouts and engine hangs trigger draining and replacement; a watchdog checks per-replica token progress.
• Overload: shed batch traffic first, then lower-priority teams per policy, and return 429s with Retry-After rather than letting latency collapse for everyone.

32–37 min Trade-offs

Build first: one pool per model with continuous batching, the gateway with quotas and metering, and least-loaded routing. Then prefix-affinity routing, batch backfill, autoscaling with warm spares, and canary rollouts.

37–40 min Wrap-up and likely follow-ups

Close with: "An OpenAI-compatible gateway with per-team quotas, a router that keeps prefix-cache affinity under bounded load, and engine pools doing continuous batching with chunked prefill, sized from the per-token latency target. Batch work backfills idle capacity and is preempted first. About fourteen 8-GPU nodes cover peak, autoscaled on queue time with warm spares, and every model change goes through a canary with quality and latency checks."

Likely follow-ups:

• How do you serve fine-tuned variants? LoRA adapters on shared base pools with multi-adapter batching; full fine-tunes as their own pools.
• How do you support 128k contexts? A separate long-context pool with larger KV budgets, fp8 KV and possibly context parallelism, so long requests don't reduce batch sizes for everyone.
• How do you charge teams? Meter input, cached-input and output tokens separately, priced to reflect cost (output tokens are memory-bound and cost more).
• How do you add speculative decoding? Enable it in low-load periods or for latency-sensitive teams; at high batch sizes it can reduce throughput.

What interviewers listen for

• Separating TTFT from time per token and deriving batch limits from the SLO.
• Capacity estimated from decode and prefill separately.
• Cache-aware routing with load bounds.
• Autoscaling that accounts for slow cold starts.
• Safe rollouts and honest handling of mid-stream failures.

Red flags

• Sizing the fleet from peak FLOPs alone.
• Scaling on GPU utilization.
• Retrying streams transparently and duplicating output.
• No isolation between teams.