Home / Fundamentals / Transformer Architecture

The transformer by the numbers

Parameter counts, FLOPs per token for training and inference, the 6N rule, KV-cache size, MFU and worked examples you should be able to do on a whiteboard.

12 min read

Research and AI-engineering interviews often include quick estimates: how many parameters, how long training takes, how much memory a KV cache needs. These follow from a handful of formulas.

Parameters

For a decoder-only model with L layers, model width d, vocabulary V:

So a dense layer has about 12d² parameters, and

N ≈ 12 · L · d²   (+ embeddings)

Worked example: d = 4,096, L = 32, V = 32,000, untied:

12 · 32 · 4096² = 6.44B      embeddings: 2 · 32,000 · 4,096 = 0.26B      total ≈ 6.7B

That's the familiar "7B" size.

FLOPs per token

A matrix multiply of an m × n matrix by an n × p matrix costs 2mnp FLOPs (a multiply and an add). Applying N parameters to one token costs about 2N FLOPs.

The backward pass costs twice the forward because each weight matmul needs two matmuls in backward: one for the gradient with respect to the input and one for the gradient with respect to the weights.

The attention term 4·L·T·d matches the parameter term 24·L·d² at T ≈ 6d, so for most context lengths the 6N rule is a good approximation.

Training time

total FLOPs  C ≈ 6 · N · D          (D = training tokens)
time         = C / (num_GPUs × peak FLOP/s × MFU)

MFU (model FLOPs utilization) is the fraction of the hardware's peak throughput spent on useful model FLOPs. Well-tuned large runs reach roughly 35–55%.

Worked example: a 7B model on 2T tokens:

C = 6 · 7e9 · 2e12 = 8.4e22 FLOPs
1,024 GPUs × ~1e15 peak bf16 FLOP/s × 40% MFU ≈ 4.1e17 FLOP/s
time ≈ 8.4e22 / 4.1e17 ≈ 2.05e5 s ≈ 2.4 days

Memory for inference

• Weights: N × bytes per parameter. A 70B model is 140 GB in bf16, ~70 GB in 8-bit, ~35 GB in 4-bit.
• KV cache per token: 2 · L · n_kv_heads · d_head · bytes.

L = 32, n_kv_heads = 8, d_head = 128, bf16 (2 bytes):
2 · 32 · 8 · 128 · 2 = 131,072 bytes = 128 KiB per token
× 32k tokens = 4 GiB per sequence

Decode speed from memory bandwidth

At batch size 1, generating each token requires reading all weights from HBM, so decode is memory-bound:

tokens/s (batch 1) ≲ memory bandwidth / bytes of weights

A 7B model in bf16 (14 GB) on a GPU with ~3.3 TB/s of bandwidth: at most ~235 tokens/s, and less in practice. Batching reuses each weight read across many sequences, which is why serving throughput grows with batch size until compute becomes the limit.

Arithmetic intensity

Arithmetic intensity is FLOPs per byte moved. Hardware has a ridge point, peak FLOP/s ÷ bandwidth, of roughly 300 FLOPs/byte for current datacenter GPUs. Work below the ridge point is memory-bound; above it, compute-bound.

• Prefill processes many tokens per weight read (intensity ∝ prompt length): compute-bound.
• Decode at batch b has intensity ≈ b FLOPs per byte for bf16 weights (2 FLOPs per parameter per token ÷ 2 bytes per parameter): memory-bound until b reaches the hundreds.

Cheat sheet

params per layer          ≈ 12 d²
N                         ≈ 12 L d² + V d (×2 if untied)
forward FLOPs / token     ≈ 2N (+ 4 L T d attention)
training FLOPs            ≈ 6 N D
KV cache / token          = 2 · L · n_kv · d_head · bytes
decode tokens/s (bs = 1)  ≲ HBM bandwidth / weight bytes
training memory (Adam)    ≈ 16 bytes / param + activations

Interview angles

• Count a model's parameters from its config in under a minute.
• Estimate training time from N, D, GPU count and MFU.
• Size a KV cache and explain why GQA matters.
• Explain why decode is memory-bound using arithmetic intensity.