Home / Fundamentals / Post-training & Alignment

LoRA, QLoRA and parameter-efficient fine-tuning

The low-rank update and its initialization and scaling, parameter and memory arithmetic, which modules to adapt, QLoRA's 4-bit base, and serving many adapters.

11 min read

LoRA

Freeze the pre-trained weight W ∈ ℝ^{d_out × d_in} and learn a low-rank update:

h = W x + (α / r) · B A x,     A ∈ ℝ^{r × d_in},  B ∈ ℝ^{d_out × r}

• Initialization: A random (Gaussian or Kaiming), B = 0, so training starts exactly from the base model.
• Scaling: α / r keeps the update's magnitude roughly independent of rank, so the learning rate needn't be retuned when changing r (rank-stabilized LoRA uses α / √r).
• Merging: after training, W' = W + (α/r) B A can be folded into the weights: zero extra inference latency.

Arithmetic

Trainable parameters per adapted matrix: r · (d_in + d_out).

7B model (h = 4096, 32 layers, MLP width 11,008), r = 16:

q, k, v, o (4096 × 4096 each): 4 × 16 × 8192 ≈ 0.52M per layer
gate, up, down (4096 ↔ 11008): 3 × 16 × 15104 ≈ 0.73M per layer
All linear layers: ≈ 1.25M × 32 ≈ 40M parameters (~0.6% of 7B)

Memory: frozen bf16 base (14 GB) + adapters with Adam state (40M × 16 B ≈ 0.64 GB) + activations. Compared with ~112 GB of state for full fine-tuning.

Choices

• Which modules: adapting all linear layers usually beats attention-only at the same parameter budget.
• Rank: 8–64 is common; higher ranks for bigger distribution shifts (new domain, new language). LoRA learns less and forgets less than full fine-tuning; for large shifts (e.g. continued pre-training) full fine-tuning often wins.
• Learning rate: around 10× higher than full fine-tuning (e.g. 1e-4 to 2e-4).

QLoRA

Quantize the frozen base to 4-bit NormalFloat (NF4, levels placed for normally distributed weights), keep LoRA adapters in bf16, and backpropagate through the dequantized base. Additions: double quantization of the scales and paged optimizer states. A 65B model can be fine-tuned on a single 48 GB GPU, with quality close to 16-bit LoRA.

Serving many adapters

Adapters are megabytes, so one base model can serve many tenants or tasks. Batched multi-adapter kernels apply each request's adapter within one batch (S-LoRA/Punica-style), with adapters cached in GPU memory and loaded on demand.

What to remember

• h = Wx + (α/r)BAx, B initialized to zero, mergeable.
• Parameters r(d_in + d_out) per matrix; ~0.5% of the model.
• QLoRA = 4-bit frozen base + bf16 adapters.