Explain Adam and AdamW. Why does decoupled weight decay matter, and what does Adam cost in memory?

30-second answer

Adam tracks moving averages of the gradient (m) and squared gradient (v), bias-corrects them, and updates θ by η·m̂/(√v̂ + ε), giving momentum plus per-parameter step sizes. Adding L2 regularization to the gradient means the decay gets divided by √v̂, so parameters with large gradient history are decayed less; AdamW applies decay directly to the weights (θ −= ηλθ), making it uniform. LLM settings are typically β2 = 0.95, weight decay ≈ 0.1, clipping at 1.0. Adam's two fp32 moments add 8 bytes per parameter, a major part of the ~16 bytes/param training footprint.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll build up from SGD with momentum to Adam, explain bias correction, show exactly where L2 regularization goes wrong inside Adam and how AdamW fixes it, then cover hyperparameters and memory."

5–15 min Core explanation

SGD with momentum: m = μm + g; θ −= ηm. One learning rate for all parameters.

Adam:

m_t = β1 m_{t−1} + (1 − β1) g_t
v_t = β2 v_{t−1} + (1 − β2) g_t²
m̂_t = m_t / (1 − β1^t),   v̂_t = v_t / (1 − β2^t)
θ_t = θ_{t−1} − η · m̂_t / (√v̂_t + ε)

The ratio m̂/√v̂ is roughly a signal-to-noise-normalized step: parameters with large, noisy gradients take smaller steps; rarely updated parameters take relatively larger ones.
Bias correction: m and v start at zero, so early estimates are biased toward zero. Dividing by 1 − β^t corrects this; with β2 = 0.999, v would otherwise be underestimated for hundreds of steps.

15–25 min Deep dive: weight decay

L2 in Adam: add λθ to the gradient: g' = g + λθ. The decay term enters m and is then divided by √v̂. Parameters with large historical gradients have large v̂, so they're decayed less; rarely updated ones are decayed more. Regularization strength becomes coupled to gradient statistics.

AdamW (decoupled):

θ_t = θ_{t−1} − η · ( m̂_t / (√v̂_t + ε) + λ θ_{t−1} )

Every weight shrinks by the same factor (1 − ηλ) per step regardless of its gradient history. Empirically this generalizes better and makes λ and η easier to tune independently. (Because decay is multiplied by η, it also follows the learning-rate schedule.)

What not to decay: biases, normalization gains, and often embeddings; decaying norm gains toward zero is harmful.

Typical LLM settings:

Hyperparameter	Typical value	Why
β1	0.9	Standard momentum
β2	0.95	Faster-adapting second moment; more stable than 0.999 at scale
ε	1e-8 (some use 1e-5)	Too small can amplify updates for tiny-`v` parameters
Weight decay	0.1	Decoupled
Gradient clip	1.0 (global norm)	Guard against bad batches

25–32 min Memory and alternatives

Per parameter, mixed-precision AdamW training holds: bf16 weights (2 B) + bf16 gradients (2 B) + fp32 master weights (4 B) + fp32 m (4 B) + fp32 v (4 B) = 16 bytes. For a 70B model that's ~1.1 TB before activations, which is why optimizer state is sharded (ZeRO/FSDP).

Alternatives:

Adafactor: factorizes the second moment into row and column statistics; much less memory, slightly less robust.
8-bit optimizers: quantize m and v with block-wise scaling.
Lion: sign of an interpolated momentum; one state tensor.
Shampoo / SOAP / Muon: use matrix structure (preconditioning or orthogonalizing updates) and report better efficiency at some scales.

32–37 min Production example

A fine-tuning job used Adam with L2 regularization from an old config. Switching to AdamW with weight decay 0.1 (excluding norms and biases) improved validation loss and removed sensitivity to the learning rate: previously, changing η also changed effective regularization in a hard-to-predict way.

37–40 min Likely follow-ups

Why does Adam need warmup? Early v̂ estimates from few samples are noisy, causing oversized steps.
What happens if ε is too small? Parameters with tiny v (rare tokens) can take huge steps when a gradient appears, causing spikes.
Does AdamW's decay interact with the LR schedule? Yes; effective decay per step is ηλ, so it shrinks as the LR decays.

What interviewers listen for

Correct equations with bias correction.
A precise explanation of the L2-vs-decoupled difference.
The 16-bytes-per-parameter accounting.

Red flags

"AdamW is Adam with L2."
Not knowing the optimizer state's memory cost.