Home / Fundamentals / Transformer Architecture

Transformer ArchitectureSeniorQuestion 5 of 9

How do rotary position embeddings (RoPE) work, and how do you extend a model's context length with them?

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll explain why transformers need positional information, derive RoPE's relative-position property, implement it, then discuss frequency structure and context extension methods."

5–15 min Core explanation

Self-attention is permutation-equivariant, so order must be injected. Absolute schemes add a position vector to embeddings; RoPE instead rotates queries and keys by position-dependent angles.

Pair up dimensions (2i, 2i+1) of a head vector. For position m, rotate pair i by m·θ_i with θ_i = b^(−2i/d_k) (b is the base, classically 10,000):

R_m(x)_{pair i} = [[cos mθ_i, −sin mθ_i], [sin mθ_i, cos mθ_i]] · (x_{2i}, x_{2i+1})

Relative property. Rotation matrices satisfy R_mᵀ R_n = R_{n−m}, so

score(m, n) = ⟨R_m q, R_n k⟩ = qᵀ R_mᵀ R_n k = qᵀ R_{n−m} k

The attention score depends on content and on the offset n − m only. Values aren't rotated.

15–25 min Deep dive

Implementation (complex-number view: treat each pair as a complex number and multiply by e^{i·m·θ_j}):

def apply_rope(x, pos, base=10000.0):
    # x: (..., T, d_k), pos: (T,)
    d_k = x.shape[-1]
    freqs = base ** (-torch.arange(0, d_k, 2, device=x.device) / d_k)  # (d_k/2,)
    ang = pos[:, None].float() * freqs[None, :]                        # (T, d_k/2)
    xc = torch.view_as_complex(x.float().reshape(*x.shape[:-1], -1, 2))
    rot = torch.polar(torch.ones_like(ang), ang)                       # e^{i·ang}
    return torch.view_as_real(xc * rot).flatten(-2).type_as(x)

Frequency structure. Pair 0 rotates fastest (period ~2π positions) and resolves fine-grained local order; the last pairs rotate very slowly (periods of tens of thousands of positions with base 10,000) and act almost like position-independent channels. Models use different frequency bands for local syntax and long-range retrieval.

Why it works well with caching. Each key is rotated once, at its own position, when it's written to the KV cache; no per-step recomputation.

Context extension. Beyond the training length, high-frequency pairs have wrapped around many times (seen during training), but low-frequency pairs reach angles never seen. Methods:

25–32 min Trade-offs and alternatives

• ALiBi adds a head-specific linear distance penalty to scores; simple and extrapolates smoothly, but biases attention toward recent tokens.
• Learned absolute embeddings can't represent positions beyond training.
• No positional encoding (NoPE) in causal decoders can work because the causal mask leaks order, but it's less controllable.
• RoPE applied to only part of each head's dimensions ("partial rotary") is a common variant.

32–37 min Production example

A team extended an 8k-context model to 32k for document QA. Running it naively past 8k produced degraded, repetitive output. NTK-aware scaling without fine-tuning improved things; YaRN plus ~1B tokens of long-document fine-tuning gave near-baseline quality at 32k on their evaluation, with no measurable regression on short tasks.

37–40 min Likely follow-ups

• Why rotate Q and K but not V? Position should affect which tokens are attended to, not the content passed along.
• Is RoPE's long-range decay real? The expected magnitude of the dot product for random vectors decays with distance under RoPE's frequency spectrum, an inductive bias toward locality.
• How does RoPE interact with MLA? Rotation doesn't commute with absorbing up-projections, so MLA keeps a separate small RoPE key component.

What interviewers listen for

• The derivation of the relative-offset property.
• Understanding of the frequency spectrum and why extrapolation fails.
• Concrete context-extension methods and the need for fine-tuning.

Red flags

• Describing RoPE as "adding sinusoids to embeddings".
• Claiming RoPE models extrapolate to any length for free.