Pre-training & Optimization

The pre-training objective and data

Next-token prediction as maximum likelihood, cross-entropy, perplexity and bits per byte, sequence packing, and how web-scale data is filtered, deduplicated, decontaminated and mixed.

Optimizers, learning rates and batch size

Adam and AdamW with equations, why decoupled weight decay, typical LLM hyperparameters, warmup, cosine and WSD schedules, batch size and the gradient noise scale, clipping and initialization.

Scaling laws and compute-optimal training

Power-law scaling of loss, the parametric form L(N, D), the compute-optimal allocation (about 20 tokens per parameter), why Kaplan and Chinchilla disagreed, over-training for inference, and limits of scaling laws.

Numerical precision and training stability

fp32, fp16, bf16 and fp8 formats, mixed-precision training with master weights and loss scaling, and how to detect and fix loss spikes and divergence at scale.

Derive the language-modelling loss. How do cross-entropy, perplexity and bits per byte relate?

Explain Adam and AdamW. Why does decoupled weight decay matter, and what does Adam cost in memory?

Why do LLMs use learning-rate warmup, and how do cosine and warmup–stable–decay schedules compare?

What do scaling laws say about how to allocate a compute budget between model size and data?

Your large pre-training run has loss spikes. How do you diagnose and fix them?

How does mixed-precision training work? Compare fp16, bf16 and fp8.

Design the data pipeline for pre-training an LLM on web data.