Why did transformers replace RNNs and LSTMs for language modelling?

30-second answer

RNNs process tokens sequentially, so training can't parallelize across the sequence, and information between distant tokens must pass through many recurrent steps, which makes long-range dependencies hard to learn despite gating. Transformers compute all positions in parallel during training, give every pair of positions a direct one-step path through attention, and scale predictably with data and compute on GPUs. The cost is quadratic attention in sequence length and a KV cache that grows during generation, which is why state-space and linear-attention models are being revisited, often in hybrids.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "I'll compare the two on three axes, which are trainability over long ranges, parallelism on hardware and scaling, then cover what transformers gave up and why recurrent ideas are coming back."

5–15 min Core explanation

RNNs and LSTMs process tokens one at a time: h_t = f(h_{t−1}, x_t).

Sequential computation: step t needs step t−1, so training can't parallelize across the sequence. On GPUs, that wastes most of the hardware.
Long paths: information from token 1 reaches token 1,000 through 999 nonlinear steps. Gating (LSTM, GRU) eases vanishing gradients but doesn't eliminate the difficulty, and a fixed-size state is a bottleneck.

Transformers compute all positions at once during training:

Parallelism: a layer is a few large matmuls over the whole sequence; GPU utilization is high.
One-step paths: attention connects any two positions directly, so long-range dependencies are easy to represent and gradients flow well.
Scaling: performance improves predictably with parameters, data and compute (scaling laws), and the architecture is stable to scale with pre-norm and good initialization.

15–25 min Deep dive: the costs

	RNN	Transformer
Training parallelism	Sequential	Fully parallel over positions
Path length between tokens	O(T)	O(1)
Compute per layer	O(T·d²)	O(T·d² + T²·d)
Inference state	Fixed size `h`	KV cache grows with T
Per-token decode cost	O(d²)	O(d² + T·d)

So transformers pay quadratic attention during training and a growing KV cache at inference, which is why long-context serving is memory-bound.

25–32 min The comeback of recurrence

Modern state-space models and linear-attention models revisit recurrence with a twist: they're linear in the state, so training can use parallel scans or chunked matmuls (parallel over the sequence), while inference uses a fixed-size recurrent state (constant memory per token). Selective state-space models made the state update input-dependent, improving language modelling. Their weakness has been precise recall over long contexts (copying, retrieving a specific fact), so many recent designs are hybrids that interleave a few attention layers with many recurrent ones.

32–37 min Production example

For an on-device assistant with tight memory, a team evaluated a hybrid recurrent/attention model against a same-size transformer. The hybrid's constant-size state allowed much longer conversations within the memory budget, and matched the transformer on most tasks, but lagged on tasks requiring exact recall of earlier details; they kept a few attention layers to close the gap.

37–40 min Likely follow-ups

Could an RNN match a transformer with enough compute? It depends on whether training can use the hardware efficiently; parallelizable recurrences are why recent recurrent models are competitive.
What does a transformer lose at inference? A growing KV cache and per-token cost that rises with context.
What inductive biases differ? RNNs bias toward recency; transformers have little built-in bias beyond positional encoding.

What interviewers listen for

Parallelism and path length as the core reasons, not just "attention is better".
Honest costs: quadratic attention, KV-cache growth.
Awareness of state-space and hybrid models.

Red flags

"RNNs can't model long sequences at all."
No mention of hardware efficiency.