Home / Fundamentals / Post-training & Alignment

Post-training & AlignmentMidQuestion 2 of 7

How does LoRA work? Work through its parameter and memory savings, and explain QLoRA.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll give the formulation and why low rank is plausible, the initialization and scaling details, compute parameter and memory savings, show code, then QLoRA and serving."

5–15 min Core explanation

Fine-tuning updates are empirically low in "intrinsic dimension": the change needed to adapt a large model lies close to a low-dimensional subspace. LoRA constrains each weight update to rank r:

h = W x + (α / r) · B (A x)
W frozen (d_out × d_in);  A: r × d_in (random init);  B: d_out × r (zero init)

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=16, alpha=32, dropout=0.05):
        super().__init__()
        self.base = base.requires_grad_(False)
        self.A = nn.Parameter(torch.empty(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale, self.drop = alpha / r, nn.Dropout(dropout)

    def forward(self, x):
        return self.base(x) + self.scale * (self.drop(x) @ self.A.T @ self.B.T)

    def merge(self):
        self.base.weight.data += self.scale * (self.B @ self.A)

• B = 0 means the model is unchanged at step 0.
• α / r scaling keeps the effective update size stable as r changes.

15–25 min Deep dive: arithmetic

Per adapted matrix: r · (d_in + d_out) parameters.

Llama-2-7B shape (h = 4096, 32 layers, MLP 11,008), r = 16, all linear layers:

attention q, k, v, o: 4 × 16 × (4096 + 4096) = 524k
MLP gate, up, down:   3 × 16 × (4096 + 11008) = 725k
per layer ≈ 1.25M  × 32 layers ≈ 40M trainable (~0.6%)

Memory comparison:

So a 7B LoRA fine-tune fits on one 24–48 GB GPU, and QLoRA fits larger models: the QLoRA paper fine-tuned a 65B model on a single 48 GB GPU.

QLoRA details: NF4 (4-bit levels at quantiles of a normal distribution), double quantization (quantizing the per-block scales too), paged optimizers to survive memory spikes; compute happens in bf16 after dequantizing each block on the fly.

25–32 min Trade-offs and choices

• LoRA learns less and forgets less than full fine-tuning. For style, format and narrow tasks it matches full fine-tuning; for large shifts (new language, continued pre-training on a new domain, heavy code), full fine-tuning tends to win.
• Modules: all linear layers > attention-only at equal budget.
• Rank: 8–64; increasing rank beyond what the data supports doesn't help.
• LR: ~10× the full fine-tuning LR.
• Serving: merge for one task (no overhead), or keep adapters separate to serve many tenants on one base with batched multi-adapter kernels; adapters are ~80 MB for the 7B example in bf16.

32–37 min Production example

A SaaS company needed per-customer writing styles for 300 customers. Full fine-tunes would mean 300 model copies. Instead they trained an r = 16 LoRA per customer (~80 MB each) and served them all from one base model per GPU pool with multi-adapter batching, loading adapters on demand with an LRU cache. Cost was close to serving the base model alone.

37–40 min Likely follow-ups

• Why initialize B to zero and not A? One of them must be zero to start from the base model; with A random, gradients flow to B from the first step.
• What's DoRA? Decomposes weights into magnitude and direction and applies LoRA to the direction; often slightly better.
• Does merging change quality? No, it's algebraically identical (up to rounding); with QLoRA, merging into a 4-bit base requires care.

What interviewers listen for

• Correct formulation with init and scaling.
• Parameter and memory arithmetic.
• When LoRA is not enough.

Red flags

• "LoRA reduces inference cost" (it doesn't, unless serving many adapters on one base).
• Not knowing what's frozen.