What is a token, and why do LLMs use subword tokenization instead of words or characters?

30-second answer

A token is an integer ID from a fixed vocabulary, learned before pre-training. Word vocabularies explode in size and can't represent unseen words; characters make sequences 4–5× longer, which is expensive for attention and decoding. Subword schemes like byte-level BPE give frequent strings single tokens and compose rare ones from pieces, with byte fallback so any input is representable. Vocabulary size trades sequence length against embedding and softmax cost, and tokenization explains several model quirks: spelling, arithmetic and uneven cost across languages.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask whether the interviewer wants the concept, the algorithm, or the consequences for modelling and cost. A strong framing: "I'll define tokens, justify subwords with the vocabulary-size vs. sequence-length trade-off, walk through byte-level BPE, then cover how tokenization shows up in model behaviour and cost."

5–15 min Core explanation

A token is an integer ID in a fixed vocabulary learned before pre-training. The model embeds token IDs, predicts a distribution over the next token ID, and never sees raw characters.

Choice	Vocabulary	Sequence length	Problem
Words	Millions	Short	Out-of-vocabulary words; huge, sparsely trained embedding and softmax
Bytes / characters	256	~4–5× longer	Quadratic attention and per-token decode cost; model must learn word formation
Subwords	32k–256k	Moderate	The practical middle ground

Byte-level BPE:

Start from the 256 bytes, after a regex pre-tokenization that splits text into words, numbers, punctuation and whitespace runs (merges never cross these).
Count adjacent symbol pairs, weighted by word frequency; merge the most frequent pair into a new symbol; record the merge.
Repeat until the target vocabulary size.
To encode, apply merges in learned order.

Because the base alphabet is bytes, any string is representable: no <unk>.

15–25 min Deep dive

Vocabulary size, quantified. For d = 4,096:

V	Embedding params (V·d)	Logit FLOPs per token (2·V·d)
32k	131M	0.26 GFLOP
128k	524M	1.05 GFLOP
256k	1.05B	2.1 GFLOP

Against a 7B model's ~14 GFLOPs per token, a 256k vocabulary makes the output layer a noticeable fraction of compute, but shortens sequences, often by 10–20% or more for code and non-English text, which saves attention, KV-cache memory and decode steps. Rare tokens receive few gradient updates, so very large vocabularies need enough data per token.

Alternatives:

WordPiece picks merges that maximize the likelihood gain of the training data.
Unigram LM (SentencePiece) prunes a large candidate vocabulary by likelihood and defines a distribution over segmentations, enabling subword regularization (sampling segmentations during training for robustness).

Pre-tokenization details matter:

Digit handling (single digits, or groups of up to three) affects arithmetic.
Whether a leading space belongs to the word (" the" ≠ "the").
Case and Unicode normalization (NFC vs. NFKC).

25–32 min Trade-offs and failure modes

Fertility (tokens per word) varies by language; under-represented scripts may cost 2–4× more tokens per sentence, raising cost and shrinking effective context.
Glitch tokens: strings frequent in the tokenizer's corpus but rare in the model's training data get untrained embeddings and can trigger odd outputs.
Character-level tasks (spelling, counting letters, reversing strings) are hard because the model sees multi-character tokens.
Tokenizer-free models (byte-level with hierarchical or patch-based architectures) avoid these artefacts but must handle longer sequences efficiently; an active research area.

32–37 min Production example

A team serving English and Japanese users found Japanese conversations hit the context limit about 2.5× sooner and cost more per message. Budgets, rate limits and truncation were switched from characters to tokens counted with the model's tokenizer, history summarization triggered earlier for high-fertility languages, and the cost model was split by locale.

37–40 min Likely follow-ups

Can you change a trained model's tokenizer? Not cheaply: embeddings are tied to IDs. You'd initialize new embeddings (e.g. as averages of old sub-pieces) and continue training.
Why do some tokenizers split digits individually? It gives a consistent place-value representation and measurably improves arithmetic.
What are special tokens and why are they a security concern? Reserved IDs for roles and tool calls; user text must never be tokenized into them, or users can forge system turns.

What interviewers listen for

The vocabulary vs. sequence-length trade-off, with numbers.
BPE explained concretely, including byte fallback and pre-tokenization.
Connecting tokenization to model behaviour, cost and multilingual fairness.

Red flags

"The model reads words."
Not knowing that cost and context limits are measured in tokens.