Home / LLM Fundamentals

Training, fine-tuning and alignment

Pre-training, supervised fine-tuning, RLHF and DPO, LoRA, and how to choose between prompting, RAG and fine-tuning.

10 min read

The training pipeline

  1. Pre-training. Next-token prediction on trillions of tokens of text and code. Loss is cross-entropy over the vocabulary. The model learns language, facts and reasoning patterns, but is not yet a helpful assistant.
  2. Supervised fine-tuning (SFT). Train on curated prompt → ideal response pairs so the model follows instructions and adopts a chat format.
  3. Preference tuning / alignment. Teach the model which of several responses people prefer.
  4. RLHF: train a reward model on human comparisons, then optimize the policy against it with an RL algorithm such as PPO, with a KL penalty to stay close to the SFT model.
  5. DPO (Direct Preference Optimization): skip the separate reward model and optimize directly on preference pairs with a classification-style loss. Simpler and more stable.
  6. RLAIF / constitutional approaches: use AI feedback guided by written principles to scale up preference data.
  7. Reasoning-focused RL. Recent models are further trained with reinforcement learning on tasks with checkable answers (math, code), which encourages longer step-by-step reasoning before answering.

Scaling laws

Loss falls predictably as a power law in parameters, data and compute. The Chinchilla result suggested compute-optimal training uses roughly 20 tokens per parameter. In practice many models are trained on far more tokens than that, because a smaller model that is cheaper to serve is worth extra training compute.

Parameter-efficient fine-tuning

Full fine-tuning updates every weight and needs optimizer state for all of them. PEFT methods train a small number of extra parameters instead.

  • LoRA: freeze W and learn a low-rank update ΔW = BA, where A is r × d and B is d × r with small r (for example 8–64). Adapters are megabytes, can be swapped per customer, and merge into the base weights for zero inference overhead.
  • QLoRA: LoRA on top of a 4-bit quantized base model, which makes fine-tuning large models feasible on one GPU.

Prompting vs. RAG vs. fine-tuning

This is one of the most common interview questions. A good answer is a decision process, not a single choice.

Need Best first tool
Better instructions, format, tone Prompting (cheapest, fastest to iterate)
Fresh, private or citable knowledge RAG
Consistent style, domain-specific behaviour, shorter prompts at scale Fine-tuning
Teach a new skill the base model lacks Fine-tuning, plus evals to prove it helped

Fine-tuning is poor at injecting facts reliably and makes updates slow. RAG keeps knowledge current and auditable. They combine well: fine-tune for behaviour, retrieve for knowledge.

Inference-time knobs

  • Temperature scales logits before softmax. Lower is more deterministic.
  • Top-k / top-p (nucleus) restrict sampling to the most likely tokens.
  • Max tokens and stop sequences bound output length.
  • Quantization (INT8, INT4, FP8) shrinks memory and speeds up decoding at a small quality cost.
  • Speculative decoding uses a small draft model to propose several tokens that the large model verifies in one pass, cutting latency without changing the output distribution.

Hallucination

Models produce fluent text that is not grounded, because pre-training rewards plausible continuations, not truth. Mitigations: grounding with retrieval, asking for citations and verifying them, lower temperature for factual tasks, letting the model say "I don't know", and evaluating faithfulness explicitly.