Home / Fundamentals / Transformer Architecture

Tokenization and embeddings

How text becomes integer IDs and vectors: BPE from scratch, byte-level fallback, vocabulary-size trade-offs, tied embeddings and tokenizer pathologies.

12 min read

A language model never sees text. It sees a sequence of integer token IDs from a fixed vocabulary, looks each one up in an embedding matrix, and at the end produces a probability distribution over the same vocabulary. Almost every practical property of an LLM, from cost and context limits to odd failures on arithmetic and spelling, is shaped by this step.

Why subwords

There are three natural choices for the unit of text:

Subword tokenizers give frequent strings their own token and compose rare strings from pieces, so nothing is ever out of vocabulary.

Byte-pair encoding from scratch

BPE learns a vocabulary by repeatedly merging the most frequent adjacent pair of symbols:

from collections import Counter

def train_bpe(corpus_words, num_merges):
    # Each word starts as a tuple of bytes; counts are word frequencies.
    words = Counter(tuple(w.encode("utf-8")) for w in corpus_words)
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for word, freq in words.items():
            for a, b in zip(word, word[1:]):
                pairs[(a, b)] += freq
        if not pairs:
            break
        best = max(pairs, key=pairs.get)
        merges.append(best)
        new_symbol = best                       # represent the merged pair as a tuple
        merged = Counter()
        for word, freq in words.items():
            out, i = [], 0
            while i < len(word):
                if i + 1 < len(word) and (word[i], word[i + 1]) == best:
                    out.append(new_symbol); i += 2
                else:
                    out.append(word[i]); i += 1
            merged[tuple(out)] += freq
        words = merged
    return merges

Encoding new text applies the learned merges in the order they were learned. Real implementations are much faster (priority queues, incremental pair counts), but the logic is the same.

Byte-level BPE starts from the 256 byte values instead of Unicode characters, so any string, in any language or containing emoji or binary junk, can be encoded without an unknown token.

Pre-tokenization splits text before BPE runs, typically with a regular expression that separates words, numbers, punctuation and whitespace. Merges never cross these boundaries. Choices here matter a lot: some tokenizers split numbers into individual digits or groups of up to three digits, which measurably helps arithmetic; others treat leading spaces as part of the word (" the" and "the" are different tokens).

Alternatives:

• WordPiece (BERT) chooses merges that most increase the likelihood of the training data rather than raw frequency.
• Unigram LM (SentencePiece) starts from a large candidate vocabulary and prunes tokens that least reduce likelihood. Because it defines a probability over segmentations, it supports subword regularization: sampling different segmentations during training as data augmentation.

Vocabulary size is a real trade-off

• Larger vocabulary → shorter sequences. Fewer tokens per document means cheaper attention, more text per context window, and fewer decode steps.
• Larger vocabulary → more parameters and compute at the ends. The embedding table and the output projection each hold V × d parameters. With V = 128k and d = 4,096 that's ~524M parameters per matrix, about 7% of a 7B model if untied. Computing logits costs 2·d·V FLOPs per token.
• Larger vocabulary → rarer tokens. Tokens that appear rarely in training get poorly trained embeddings. "Glitch tokens", strings that were frequent in the tokenizer's training data but rare in the model's, can produce bizarre behaviour.

Recent models moved from ~32k toward 100k–256k vocabularies, largely to serve multilingual text and code efficiently.

Fertility: not every language costs the same

Fertility is the average number of tokens per word. A tokenizer trained mostly on English may need 1.3 tokens per English word but 3–4× more for languages written in other scripts. Those users pay more, hit context limits sooner, and get slower responses. Multilingual tokenizer training and larger vocabularies narrow the gap.

Embeddings and the output layer

The embedding lookup is equivalent to multiplying a one-hot vector by the embedding matrix E ∈ ℝ^{V×d}. At the output, the final hidden state h ∈ ℝ^d is mapped to logits over the vocabulary:

logits = h · Wᵀ        W ∈ ℝ^{V×d}
p(next token) = softmax(logits)

Weight tying sets W = E: the same matrix embeds inputs and scores outputs. It saves V·d parameters and often helps smaller models; many large models leave them untied for extra capacity.

Special tokens and chat templates

Chat models wrap messages in special tokens that mark roles, turns and tool calls. Two practical rules:

• User-supplied text must never be able to produce special tokens. Tokenize user content with special-token parsing disabled, or an attacker can forge a "system" turn.
• Always use the model's own chat template. Small differences in whitespace or role markers measurably change behaviour.

Pathologies worth knowing

• Spelling and counting letters: the model sees strawberry as a few subword tokens, not ten letters, so character-level questions require recalling spelling indirectly.
• Arithmetic: inconsistent digit chunking (12345 as 123 + 45) makes place value hard to learn.
• Trailing whitespace: a prompt ending in a space can force an unusual token boundary and degrade the next token, which is why some libraries "heal" the last token.
• Token boundaries in constrained decoding: grammars defined on characters must be mapped carefully onto tokens.

Interview angles

• Explain why subwords beat words and characters, using the vocabulary size vs. sequence length trade-off.
• Walk through BPE merges on a toy corpus.
• Connect tokenization to cost, context limits, multilingual fairness and failure modes.
• Know that changing a trained model's tokenizer requires re-learning embeddings and usually continued training.