LLM FundamentalsSeniorQuestion 12 of 13

What is a Mixture-of-Experts model?

30-second answer

MoE replaces the dense feed-forward layer with many expert FFNs and a learned router that sends each token to a small number of them (e.g. top-2). Total parameters grow a lot while per-token compute stays modest. Challenges include load balancing across experts (auxiliary losses), higher memory needs since all experts must be resident, and more complex distributed serving.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Framing: "MoE decouples total parameters from per-token compute. I'll explain the routing layer, the training challenges, and the serving implications, which are often what interviewers care about."

5–15 min Core explanation

In a dense Transformer, every token passes through the same feed-forward network (FFN) in each layer. An MoE layer replaces that FFN with:

E expert FFNs (e.g. 8, 64 or more), and
a router (a small linear layer + softmax) that scores experts for each token.

Each token is sent to the top-k experts (often k = 1 or 2, sometimes more fine-grained). Outputs are combined, weighted by router scores.

Result: a model can have, say, 400B total parameters but only ~20–40B active per token. Training and inference FLOPs scale with active parameters; capacity scales with total parameters.

15–25 min Deep dive: training challenges

Load balancing: routers tend to collapse onto a few experts. An auxiliary loss encourages even usage; newer approaches use bias adjustments instead of auxiliary losses.
Capacity factor: each expert has a per-batch token limit; overflow tokens are dropped or rerouted. Too low drops tokens; too high wastes compute.
Stability: router logits can explode; router z-loss and careful initialization help.
Shared experts: some designs keep one always-active expert for common knowledge plus many routed ones.
Specialization: experts often specialize on token types or syntax rather than clean human topics.

25–32 min Serving trade-offs

Aspect	Dense	MoE
FLOPs per token	All params	Active params only
Memory	All params	All params must be resident
Batch efficiency	Predictable	Tokens scatter across experts; small batches use experts poorly
Parallelism	Tensor/pipeline	Plus expert parallelism with all-to-all communication

MoE is attractive at high throughput where batching keeps experts busy, and on clusters with fast interconnect. For small, latency-sensitive single-user deployments, memory requirements can make MoE less attractive than a dense model with the same active size.

32–37 min Production example

A team comparing a dense 70B model with an MoE of ~45B active / ~300B total found the MoE matched quality with lower per-token compute, but needed more GPUs to hold the weights. At their high request volume, the throughput gain outweighed the extra memory; for a low-volume on-prem customer they shipped the dense model instead.

37–40 min Likely follow-ups

What's top-k routing's effect on quality vs. speed? Higher k improves quality at proportional compute cost.
Can you fine-tune MoE models? Yes, but load balancing must be maintained; LoRA can target experts or shared layers.
Why is all-to-all communication expensive? Tokens move between devices hosting different experts every MoE layer.

What interviewers listen for

Active vs. total parameters.
Load balancing and capacity factors.
Memory and communication costs in serving.

Red flags

"MoE uses less memory."
Assuming experts map to human topics.