Estimate the parameters, training FLOPs and training time for a 7B-parameter transformer.

30-second answer

Each dense layer has about 12d² parameters (4d² attention, 8d² MLP), so N ≈ 12·L·d² plus V·d per embedding matrix; with d = 4,096, L = 32, V = 32k that's ≈ 6.7B. Training costs ≈ 6N FLOPs per token (2N forward, 4N backward), so 2T tokens need ≈ 8.4e22 FLOPs. On 1,024 GPUs at ~1e15 peak FLOP/s and 40% MFU that's ≈ 2.4 days. Inference weights are 2 bytes per parameter in bf16, and the KV cache is 2·L·n_kv·d_head·bytes per token.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask for (or assume) the configuration: width d, layers L, vocabulary V, attention type, MLP type, tokens D, hardware. Framing: "I'll count parameters layer by layer, derive the 6N rule for training FLOPs, then estimate time on a cluster and inference memory."

5–15 min Core explanation

Per layer (dense, MHA, 4× MLP or SwiGLU with ~8d/3 hidden):

attention: W_Q, W_K, W_V, W_O         4 d²
MLP:       2 · d · 4d  (or 3 · d · 8d/3)   8 d²
norms:     ~2 d                         (negligible)
total ≈ 12 d²

Whole model: N ≈ 12·L·d² + V·d (+ another V·d if the output head is untied).

Example (d = 4,096, L = 32, V = 32,000, untied):

12 · 32 · 16.78M = 6.44B   +   2 · 32,000 · 4,096 = 0.26B   ≈ 6.7B

With GQA (8 KV heads of 32), attention drops from 4d² to 2d² + 2d²·8/32 = 2.5d²; models usually compensate with a slightly larger MLP.

15–25 min Deep dive: FLOPs

A matmul (m × n)·(n × p) costs 2mnp FLOPs. Each parameter participates in one multiply-add per token in the forward pass, so the forward pass costs ≈ 2N FLOPs per token, plus attention's 4·L·T·d.

The backward pass computes, for every weight matmul Y = XW, both dX = dY·Wᵀ and dW = Xᵀ·dY: two matmuls of the same size, so ≈ 4N. Total: ≈ 6N FLOPs per training token, or ~8N with full activation recomputation.

Training compute and time:

C = 6 · N · D = 6 · 6.7e9 · 2e12 ≈ 8.0e22 FLOPs
throughput = GPUs × peak × MFU = 1,024 × 989e12 × 0.40 ≈ 4.05e17 FLOP/s
time ≈ 8.0e22 / 4.05e17 ≈ 2.0e5 s ≈ 2.3 days

Memory for training (mixed precision, Adam): ~16 bytes per parameter for weights, gradients and optimizer state (bf16 weights 2 + bf16 grads 2 + fp32 master 4 + Adam m 4 + Adam v 4), so ~107 GB for 6.7B, plus activations. That's why sharding (ZeRO/FSDP) is needed even for a 7B model on 80 GB GPUs.

25–32 min Inference numbers

weights (bf16):           2 · 6.7e9 ≈ 13.4 GB
KV cache per token:       2 · L · n_kv · d_h · 2 bytes
   MHA (32 heads × 128):  2 · 32 · 32 · 128 · 2 = 512 KiB
   GQA (8 KV heads):      128 KiB
decode bound (batch 1):   3.35 TB/s ÷ 13.4 GB ≈ 250 tokens/s upper bound

Real decode is slower (kernel overheads, KV reads), and batching raises total throughput because weights are read once per step for the whole batch.

32–37 min Production example

Before committing GPUs for a continued-pre-training project, a team estimated 6N·D for 7B on 50B tokens (~2.1e21 FLOPs): about 1.5 hours on 1,024 GPUs at 40% MFU, or about 3.8 days on 16 GPUs. Measured MFU on their stack was 32%, so they budgeted 5 days, which matched the actual run within half a day.

37–40 min Likely follow-ups

What's MFU vs. HFU? Model FLOPs utilization counts only the model's required FLOPs; hardware FLOPs utilization also counts recomputation.
How does MoE change the count? Compute scales with active parameters; memory scales with total parameters.
When does the attention term matter? At contexts comparable to or larger than ~6d, or when estimating long-context fine-tuning.

What interviewers listen for

Fast, correct parameter counting from a config.
A derivation of 6N, not just recall.
Turning FLOPs into wall-clock time with MFU, and memory into GPU counts.

Red flags

Forgetting the backward pass or the optimizer state.
Treating peak FLOPs as achievable.