What do scaling laws say about how to allocate a compute budget between model size and data?

30-second answer

Loss follows L(N, D) = E + A/N^α + B/D^β, and training compute is C ≈ 6ND. Minimizing L subject to the budget gives N_opt ∝ C^{β/(α+β)} and D_opt ∝ C^{α/(α+β)}; with α ≈ β both grow as about √C, roughly 20 tokens per parameter (Chinchilla: 70B on 1.4T tokens beat a 280B model at similar compute). At 1e24 FLOPs that's ~90B parameters on ~1.8T tokens. Earlier work favoured bigger models largely because schedules weren't matched to run length. Compute-optimal isn't inference-optimal: widely served models are deliberately trained far longer than 20 tokens per parameter.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask whether the goal is training-compute-optimal or total-cost-optimal (including inference). Framing: "I'll state the parametric law, derive the optimal split under C = 6ND, give the ~20 tokens/parameter rule with a worked example, explain why earlier results differed, and then adjust for inference and data limits."

5–15 min Core explanation

L(N, D) = E + A · N^(−α) + B · D^(−β)        C ≈ 6 N D

Minimize L subject to fixed C. Substituting D = C/(6N) and differentiating:

α A N^(−α) = β B D^(−β)
N_opt = G · (C/6)^(β/(α+β)),   D_opt = G⁻¹ · (C/6)^(α/(α+β)),   G = (αA / βB)^(1/(α+β))

Empirically α ≈ β, so N_opt and D_opt both scale roughly as C^0.5. The Chinchilla analysis (several methods: fixing model size and varying tokens, IsoFLOP profiles, and a parametric fit) concluded ≈ 20 tokens per parameter at compute-optimality.

15–25 min Deep dive

Scroll sideways to see the whole diagram.

IsoFLOP curves (illustrative). Each curve fixes training compute C and varies model size N, with tokens D = C / 6N. Too small a model is capacity-limited; too large a model sees too few tokens. The minima move right as compute grows, at roughly 20 tokens per parameter. Curves are computed from a parametric loss fit and are illustrative, not measured.

IsoFLOP profiles. For several fixed budgets, train models of different sizes (each with D = C/6N) and plot final loss against N. Each curve is U-shaped: too small a model is capacity-limited, too large a model sees too little data. Fit the minima across budgets to get N_opt(C).

Worked example. With 20 tokens per parameter, C = 120 N²:

C = 5.9e23  →  N ≈ 70B,  D ≈ 1.4T     (Chinchilla's configuration)
C = 1e24    →  N ≈ 91B,  D ≈ 1.8T
C = 1e25    →  N ≈ 290B, D ≈ 5.8T

Why the earlier law (N ∝ C^0.73) was different. Learning-rate schedules weren't matched to each run's length, so shorter-data runs looked worse than they were and the fit favoured bigger models with fewer tokens. Choices about counting embedding parameters and fitting at small scales also mattered. The parametric fits are fragile: later re-analyses re-estimated the constants, but the ~20 tokens/parameter conclusion held.

25–32 min Adjusting for reality

Inference costs. Total cost ≈ training (6ND) + inference (2N per generated token × lifetime tokens). If a model will serve trillions of tokens, a smaller N trained on many more tokens minimizes total cost, even though its training is "over-trained" relative to Chinchilla. This is why popular open models train with hundreds to thousands of tokens per parameter.

Data limits. Unique high-quality tokens are finite. Repeating data up to ~4 epochs is nearly as good as fresh data; beyond that, value decays. Better filtering, code and math data, multilingual data and synthetic data stretch the supply.

What laws don't give you.

Loss, not capabilities: task metrics can look abrupt (partly an artefact of discontinuous metrics like exact match).
A fixed recipe: architecture, data and optimizer changes shift the curves. Use scaling studies to compare recipes by their curves, not single points.

32–37 min Production example

A team with a fixed GPU budget for a model that would be served at high volume compared a compute-optimal 30B model with a 13B model trained on ~3× more tokens. Small-scale fits predicted a slightly higher loss for the 13B, which evaluations confirmed was within noise on their target tasks, while inference cost was less than half. They shipped the 13B.

37–40 min Likely follow-ups

How do you run a scaling study cheaply? Several small models, WSD schedules with branched decays, fit the law, extrapolate one or two orders of magnitude, and validate with one mid-size run.
Do scaling laws apply to fine-tuning or RL? Similar power laws appear in many settings, but with different exponents and stronger dependence on data quality.
What about MoE? Laws are often expressed in active parameters and total parameters separately.

What interviewers listen for

The derivation from L(N, D) and C = 6ND.
Quick arithmetic with 20 tokens per parameter.
Training-optimal vs. inference-optimal reasoning.
Awareness of methodological pitfalls.

Red flags

"Bigger models are always better for a fixed budget."
Treating fitted constants as precise physical laws.