What is the KV cache? How large does it get, and how do you reduce it?

30-second answer

In autoregressive decoding the keys and values of earlier tokens never change, so they're cached and each step only computes projections for the new token and attends over the cache, turning quadratic recomputation into linear work per step. It costs 2 × layers × KV heads × head dim × bytes per token: ~320 KB for a 70B GQA model in bf16 (1.3 GB per 4k-token sequence), 8× more with full multi-head attention. It caps batch size and context length, so it's reduced with GQA/MQA or latent attention, fp8/int8 KV, paged allocation, prefix sharing and sliding-window layers.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll explain what's cached and why it's valid, derive the size formula with numbers, explain how it limits serving, then go through the ways to shrink or manage it."

5–15 min Core explanation

In a causal transformer, token t's keys and values at each layer depend only on tokens ≤ t. When generating token t+1, everything computed for earlier positions is unchanged. Without caching, each step would recompute all previous positions: O(T²) total work for T tokens. With caching, each step computes Q, K, V only for the new token, appends K and V to the cache, and attends over it.

# one decode step, one layer (single head for clarity)
q, k, v = x_new @ Wq, x_new @ Wk, x_new @ Wv       # (1, d_h)
K_cache = torch.cat([K_cache, k]); V_cache = torch.cat([V_cache, v])
attn = softmax(q @ K_cache.T / sqrt(d_h)) @ V_cache  # (1, d_h)

Size:

bytes per token = 2 × L × n_kv × d_head × bytes_per_value

70B-class model: L = 80, d_head = 128, bf16.

Attention	n_kv	Per token	4k tokens	32 × 4k
MHA	64	2.6 MB	10.7 GB	343 GB
GQA	8	320 KB	1.3 GB	43 GB
MQA	1	40 KB	0.16 GB	5.2 GB

15–25 min Deep dive: why it matters for serving

Capacity: on 8 × 80 GB GPUs, weights take 140 GB; the remaining ~450 GB of KV space holds ~340 sequences of 4k tokens with GQA, but only ~40 with MHA. Batch size, and therefore throughput, is bounded by KV memory.
Bandwidth: every decode step reads each sequence's entire cache. At long contexts, KV reads exceed weight reads: one 32k-token GQA sequence is ~10 GB, comparable to the per-GPU weight shard.
Long context: at 128k tokens, a single GQA sequence needs ~42 GB.

25–32 min Reducing and managing it

Technique	Saving	Cost
GQA / MQA	n_heads / n_kv (8× for 64→8)	Small quality cost; must be trained in (or uptrained)
Multi-head latent attention	Stores a compressed latent per token	Architecture change; extra projection compute
fp8 / int8 KV	2× vs bf16	Small quality risk; needs scales
Sliding-window / local layers	Bounded cache for those layers	Long-range information must flow through global layers
Paged allocation	Removes fragmentation and over-reservation	Kernel and scheduler complexity
Prefix sharing	Shared system prompts/documents stored once	Cache management
Eviction (e.g. keep attention sinks + recent tokens)	Bounded memory	Can lose information; quality risk
Offload to CPU / remote storage	Very long or idle sessions	Transfer latency

32–37 min Production example

A team serving a 13B MHA model at 8k contexts could fit only ~12 concurrent sequences per GPU, so throughput was poor. Switching to an fp8 KV cache doubled concurrency; moving to a GQA model of similar quality multiplied it again by ~5. Prefix caching for the shared 1,500-token system prompt cut TTFT by ~40%.

37–40 min Likely follow-ups

Why not cache queries? A query is used only once, by its own position.
What about the attention computation cost? Each decode step is still O(T) per layer; the cache avoids recomputing K/V, not attending over them.
How does the cache interact with RoPE? Keys are cached after rotation, so positions are baked in; reusing a prefix at a different offset needs care.

What interviewers listen for

The formula with real numbers.
KV memory as the batch-size limit, and KV reads as a bandwidth cost.
Several reduction techniques with trade-offs.

Red flags

Saying the KV cache makes attention O(1).
Not knowing how GQA affects cache size.