Transformer Architecture

Tokenization and embeddings

How text becomes integer IDs and vectors: BPE from scratch, byte-level fallback, vocabulary-size trade-offs, tied embeddings and tokenizer pathologies.

Self-attention from first principles

Scaled dot-product attention with shapes and code, why √d_k, masking, multi-head attention, FLOP accounting, FlashAttention's online softmax, and MQA/GQA/MLA.

Positions, normalization and the residual stream

Sinusoidal, learned, RoPE and ALiBi positions; context extension; LayerNorm vs RMSNorm; pre-norm vs post-norm; SwiGLU; and the residual-stream view of a transformer.

The transformer by the numbers

Parameter counts, FLOPs per token for training and inference, the 6N rule, KV-cache size, MFU and worked examples you should be able to do on a whiteboard.

What is a token, and why do LLMs use subword tokenization instead of words or characters?

Explain scaled dot-product attention, implement it, and justify dividing by √d_k.

Compare MHA, MQA, GQA and MLA. What problem do they solve?

Why is attention O(n²), how does FlashAttention help, and how do models handle long context?

How do rotary position embeddings (RoPE) work, and how do you extend a model's context length with them?

Why do modern LLMs use pre-norm and RMSNorm? What are the trade-offs?

Estimate the parameters, training FLOPs and training time for a 7B-parameter transformer.

How does a Mixture-of-Experts transformer work, and what are its training and serving challenges?

Why did transformers replace RNNs and LSTMs for language modelling?