Post-training & Alignment

Supervised fine-tuning

How a base model becomes an assistant: chat templates, loss masking, data quality over quantity, packing, hyperparameters, forgetting, and where SFT sits in the post-training pipeline.

Reward models, RLHF and DPO

The Bradley–Terry reward model, the KL-regularized RLHF objective and PPO, deriving DPO from the optimal policy, its variants, and online versus offline preference learning.

Reinforcement learning for reasoning

Reinforcement learning with verifiable rewards, GRPO's group-relative advantages, reward hacking and length effects, test-time compute, and distilling reasoning into smaller models.

LoRA, QLoRA and parameter-efficient fine-tuning

The low-rank update and its initialization and scaling, parameter and memory arithmetic, which modules to adapt, QLoRA's 4-bit base, and serving many adapters.

How does supervised fine-tuning turn a base model into an assistant? Discuss chat templates, loss masking and data.

How does LoRA work? Work through its parameter and memory savings, and explain QLoRA.

When would you fine-tune instead of using RAG or prompting, and vice versa?

How is a reward model trained, and how do you detect and prevent reward hacking?

Compare RLHF with PPO and DPO. Derive the DPO objective.

How are reasoning models trained with reinforcement learning? Explain RL with verifiable rewards and GRPO.

Design the post-training pipeline that turns a new base model into a helpful, safe assistant with strong reasoning.