Home / Question bank / LLM Fundamentals

LLM FundamentalsJuniorQuestion 1 of 13

What is a token, and why do LLMs use subword tokenization instead of words or characters?

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Start by confirming what the interviewer wants: the concept, the algorithm (BPE in detail), or the practical consequences for cost and behaviour. A good framing sentence: "I'll define a token, explain why neither words nor characters work well, walk through BPE, and finish with the practical effects on cost, context limits and model quirks."

Assumptions worth stating:

• We're talking about a decoder-only LLM with a fixed vocabulary learned before pre-training.
• The vocabulary is typically 32k–200k entries.

5–15 min Core explanation

A token is an integer ID. The tokenizer maps text to a sequence of IDs; the model only ever sees those IDs, looks each one up in an embedding matrix, and predicts the next ID.

Why not words?

• The vocabulary would need millions of entries to cover names, typos, code identifiers and every language. Huge embedding and output matrices, most rows barely trained.
• Any unseen word becomes <UNK>, losing information entirely.

Why not characters (or bytes)?

• Sequences become 4–5× longer. Attention cost grows with the square of length, and generation takes one forward pass per token, so latency explodes.
• The model must spend capacity learning to assemble characters into meaning.

Subword tokenization sits in between: frequent strings get their own token, rare strings are composed from pieces, and nothing is ever out of vocabulary.

15–25 min Deep dive: how BPE works

Byte-Pair Encoding, as used by most modern LLMs (often in a byte-level form):

1. Start with a base vocabulary of all 256 bytes.
2. Count every adjacent pair of symbols across a large training corpus.
3. Merge the most frequent pair into a new symbol and add it to the vocabulary.
4. Repeat until the vocabulary reaches the target size. Record the merge order.

Encoding new text applies the learned merges in priority order.

corpus words:  low lower lowest newer
merges:        (l,o)->lo  (lo,w)->low  (e,r)->er  (n,e)->ne ...
"lowest"   ->  low + est
"slowness" ->  s + low + ness

Related schemes:

Pre-tokenization rules (splitting on spaces, digits, punctuation before BPE) matter a lot. For example, some tokenizers split numbers into single digits or groups of up to three, which noticeably changes arithmetic ability.

25–32 min Trade-offs and alternatives

• Vocabulary size: a larger vocab means shorter sequences (cheaper attention, faster generation) but a bigger embedding/output layer and rarer tokens that get less training. Many recent models moved from ~32k toward 100k+ to serve multilingual text and code efficiently.
• Fairness across languages: a tokenizer trained mostly on English makes other scripts 2–4× more expensive per sentence and consumes more context.
• Byte-level / tokenizer-free models avoid tokenization artefacts but pay the sequence-length cost; research continues on hierarchical byte models.

32–37 min Production example

In a support product serving English and Japanese users, the same answer cost far more tokens in Japanese. Consequences you'd plan for:

• Budget and rate-limit by tokens, not characters or requests.
• Measure context usage per language; Japanese conversations hit the context limit sooner, so summarize history earlier.
• When truncating text to fit, truncate on token counts using the model's own tokenizer, never by characters.

37–40 min Likely follow-ups

• Why can't models reliably count letters in a word? They see a few subword tokens, not letters, and must recall spelling indirectly.
• Can you change the tokenizer of a trained model? Not cheaply. Embeddings are tied to token IDs; you'd need to re-learn embeddings and usually continue training.
• What are special tokens? Reserved IDs for structure: beginning/end of sequence, chat role markers, tool-call delimiters. They shouldn't be injectable from user text.
• Rule of thumb for English? About 4 characters or ¾ of a word per token.

What interviewers listen for

• You explain why subwords: vocabulary size vs. sequence length.
• You can walk through BPE merges concretely.
• You connect tokenization to cost, latency, context limits and quirks.

Red flags

• Saying the model "reads words" or "sees characters".
• Not knowing that output tokens drive latency and cost.
• Treating token counts as interchangeable across languages.