Home / Fundamentals / Pre-training & Optimization

Scaling laws and compute-optimal training

Power-law scaling of loss, the parametric form L(N, D), the compute-optimal allocation (about 20 tokens per parameter), why Kaplan and Chinchilla disagreed, over-training for inference, and limits of scaling laws.

13 min read

Power laws

Across many orders of magnitude, pre-training loss falls as a smooth power law in model size N, data D and compute C, as long as the other two aren't the bottleneck. This predictability is what lets labs plan large runs from small experiments.

A widely used parametric form:

L(N, D) = E + A / N^α + B / D^β

E is the irreducible loss (the entropy of the data under this tokenizer).
A / N^α is the penalty for a finite model.
B / D^β is the penalty for finite data.

Fitted exponents are typically around 0.3 for both α and β.

The compute-optimal allocation

Training compute is C ≈ 6ND. For a fixed budget, choose N and D to minimize loss. Substitute D = C / (6N) and set the derivative to zero:

dL/dN = −α A N^(−α−1) + β B D^(−β) / N = 0
⇒  α A N^(−α) = β B D^(−β)
⇒  N_opt ∝ C^(β/(α+β)),    D_opt ∝ C^(α/(α+β))

With α ≈ β, both exponents are about 0.5: model size and data should grow together. The Chinchilla study found roughly 20 training tokens per parameter to be compute-optimal. Their 70B model trained on 1.4T tokens (≈ 5.9e23 FLOPs) beat a 280B model trained on 300B tokens with similar compute.

Scroll sideways to see the whole diagram.

IsoFLOP curves (illustrative). Each curve fixes training compute C and varies model size N, with tokens D = C / 6N. Too small a model is capacity-limited; too large a model sees too few tokens. The minima move right as compute grows, at roughly 20 tokens per parameter. Curves are computed from a parametric loss fit and are illustrative, not measured.

Worked example. With 20 tokens per parameter, C = 6 · N · 20N = 120 N², so N_opt = √(C / 120):

C = 1e24 FLOPs  →  N ≈ √(8.3e21) ≈ 91B parameters,  D ≈ 1.8T tokens

Why the earlier law disagreed

An earlier scaling study recommended putting most extra compute into parameters (N ∝ C^0.73), implying much larger, under-trained models. The main reasons for the difference were methodological: learning-rate schedules weren't adjusted to each run's length, so shorter runs looked artificially worse, plus choices about which parameters to count and the range of scales fitted. The parametric fits themselves are sensitive to these details; later re-analyses refit the constants but kept the ~20 tokens-per-parameter conclusion.

Compute-optimal isn't inference-optimal

Chinchilla optimizes training compute. If a model will serve billions of requests, a smaller model trained on far more tokens can have lower total cost (training plus inference), even if its training run is "inefficient". This is why many widely deployed models are trained with hundreds or thousands of tokens per parameter.

Data constraints

High-quality text is finite. Repeating data for a few epochs is nearly as good as fresh data; beyond about four epochs returns drop quickly. Options when data-constrained: careful repetition, better filtering, more code and math, multilingual data, and synthetic data, each with its own risks.

What scaling laws do and don't tell you

They predict loss, not specific capabilities. Downstream accuracy often rises smoothly with loss on average, but individual tasks can look abrupt, partly because metrics like exact match are discontinuous.
They hold for a fixed recipe: architecture, data mixture and optimizer changes shift the curves.
They're used to compare recipes (does a change improve the curve's offset or slope?), to pick hyperparameters by extrapolation, and to forecast large runs from small ones.

Interview angles

Derive the compute-optimal allocation from L(N, D) and C = 6ND.
Do the 20-tokens-per-parameter calculation quickly.
Explain why over-training smaller models is rational for deployment.
Discuss the limits: data scarcity, capability vs. loss.