Fundamentals

Tokenization, attention, positional encodings, normalization, mixture-of-experts, and the arithmetic of parameters, FLOPs and memory.

• Tokenization and embeddings12 min
• Self-attention from first principles16 min
• Positions, normalization and the residual stream15 min
• The transformer by the numbers12 min

The language-modelling objective, pre-training data, AdamW and learning-rate schedules, scaling laws, numerical precision and training stability.

• The pre-training objective and data14 min
• Optimizers, learning rates and batch size15 min
• Scaling laws and compute-optimal training13 min
• Numerical precision and training stability13 min

Training memory and compute arithmetic, data parallelism and collectives, ZeRO and FSDP, tensor, pipeline, sequence and expert parallelism, and keeping thousands of GPUs productive.

• Memory and compute arithmetic for training13 min
• Data parallelism, collectives and ZeRO/FSDP14 min
• Tensor, pipeline, sequence and expert parallelism15 min
• Reliability and efficiency at scale11 min

Roofline analysis of prefill and decode, the KV cache, continuous batching and paged attention, quantization, speculative decoding, and decoding strategies.

• The inference roofline: prefill vs decode13 min
• KV cache, continuous batching and paged attention14 min
• Quantization and speculative decoding13 min
• Decoding strategies10 min

Supervised fine-tuning, reward models, RLHF and DPO, reinforcement learning with verifiable rewards, LoRA and QLoRA, and when to fine-tune at all.

• Supervised fine-tuning12 min
• Reward models, RLHF and DPO15 min
• Reinforcement learning for reasoning13 min
• LoRA, QLoRA and parameter-efficient fine-tuning11 min

Benchmarks and contamination, pass@k, LLM-as-judge and arena ratings, the statistics of model comparisons, hallucination measurement, and building evaluation suites.

• Benchmarks, contamination and what scores mean12 min
• LLM-as-judge and preference ratings12 min
• The statistics of model comparisons11 min