Home / Question bank / LLM Fundamentals
What is the KV cache and what problem does it solve?
30-second answer
In autoregressive decoding, keys and values for earlier tokens don't change. Caching them means each new token computes only its own projections and attends to the cache, turning per-step cost from re-processing the whole sequence into a single-token step. The trade-off is memory: the cache grows with layers, KV heads, head dimension and sequence length, and often limits batch size in serving.
Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.
0–5 min Clarify and frame
Clarify the context: single-request inference or serving many requests? Framing: "I'll explain what's recomputed without a cache, how the cache fixes it, compute its memory size, then cover the serving techniques built around it."
5–15 min Core explanation
Autoregressive generation produces one token per forward pass. At step t, the new token attends over all previous tokens, which needs their keys and values in every layer.
Without a cache, each step re-runs the whole sequence through the model to recompute those K and V. That's quadratic total work across a generation.
The keys and values of past tokens never change (the model is causal), so we store them. Each new step:
- Computes Q, K, V for just the new token.
- Appends its K, V to the cache.
- Attends with its Q against all cached K, V.
Per-step compute drops from processing the full sequence to processing one token.
15–25 min Deep dive: memory math
Per token, per layer, we store one K and one V vector for each KV head:
bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_value
Example: 32 layers, 8 KV heads (GQA), head_dim 128, FP16 (2 bytes):
2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 128 KB per token
32k-token context ≈ 4 GB per sequence
With full multi-head attention (say 32 KV heads) it would be 4× larger: 16 GB per sequence. This is why the KV cache, not the weights, often limits how many concurrent requests a GPU can serve.
Two phases:
- Prefill: process the prompt in parallel, fill the cache. Compute-bound.
- Decode: one token at a time, read the whole cache each step. Memory-bandwidth-bound.
25–32 min Trade-offs and techniques
| Technique | What it does | Trade-off |
|---|---|---|
| MQA / GQA | Fewer KV heads | Small quality cost, big memory win |
| PagedAttention (vLLM) | Stores cache in fixed-size blocks like virtual memory | Avoids fragmentation, enables high batch sizes |
| KV quantization (FP8/INT8) | Smaller cache entries | Minor accuracy loss |
| Prefix / prompt caching | Reuse cache for shared prefixes across requests | Needs identical prefix |
| Sliding-window attention | Keep only recent tokens | Loses distant context |
| Offloading to CPU | Store cold cache off GPU | Adds latency |
32–37 min Production example
A chat service with a 3k-token system prompt and tool definitions shared by every request: prefix caching stores that prompt's KV once, so each request only prefills the user's new message. Time-to-first-token and cost both drop significantly. On the capacity side, the team sizes GPU fleets by concurrent tokens in cache, not requests per second, and sets max context per request to prevent a few long conversations from evicting everyone else.
37–40 min Likely follow-ups
- Why don't we cache queries? Past queries are never reused; only the newest token's query is needed.
- What happens when the cache is full? Requests are queued, preempted (swapped or recomputed), or rejected; context may be truncated.
- Does the cache change outputs? No. It's an exact optimization.
- How does beam search interact with it? Beams share prefix blocks; paged caches support copy-on-write.
What interviewers listen for
- Why caching is valid (causality: past K/V don't change).
- The memory formula and realistic numbers.
- Awareness of GQA, paged attention and prefix caching.
Red flags
- Thinking the cache stores outputs or text.
- Not realizing memory grows linearly with sequence length and batch size.