Home / Fundamentals / Pre-training & Optimization

The pre-training objective and data

Next-token prediction as maximum likelihood, cross-entropy, perplexity and bits per byte, sequence packing, and how web-scale data is filtered, deduplicated, decontaminated and mixed.

14 min read

Pre-training is simple to state: predict the next token, on trillions of tokens. Most of the difficulty is in the data and in keeping a very large optimization stable.

The objective

A language model defines a distribution over sequences by the chain rule:

p(x_1, …, x_T) = Π_t p_θ(x_t | x_<t)

Training maximizes the log-likelihood of the data, which is the same as minimizing the average cross-entropy between the data distribution and the model:

L(θ) = − (1/T) Σ_t log p_θ(x_t | x_<t)

For one position with logits z and target token y: loss = −z_y + log Σ_j e^{z_j}, and the gradient with respect to the logits is simply softmax(z) − onehot(y).

Cross-entropy decomposes as H(p, q) = H(p) + KL(p ‖ q). The data's own entropy H(p) is an irreducible floor, which is the E term in scaling laws; training can only shrink the KL part.

Teacher forcing: during training the model always conditions on the true previous tokens, so every position in a sequence is trained in parallel with one forward pass and a causal mask.

Perplexity and bits per byte

Perplexity = exp(mean negative log-likelihood per token). A perplexity of 8 means the model is, on average, as uncertain as a uniform choice among 8 tokens.
Perplexity depends on the tokenizer: models with different vocabularies aren't comparable.
Bits per byte normalizes by the raw text length instead: BPB = (total NLL in nats / ln 2) / (number of UTF-8 bytes). It allows comparison across tokenizers.

Sequence packing

Documents vary in length, but training uses fixed-length sequences (e.g. 4,096 or 8,192 tokens). Documents are concatenated with end-of-sequence tokens and cut into chunks. By default, tokens can attend across document boundaries within a chunk; many recipes add a document mask so attention stays within each document, which helps especially for long-context training.

Where the data comes from

A typical mixture: filtered web crawl (the bulk), code, books, scientific papers, encyclopedic text, mathematics, multilingual data, and increasingly curated or synthetic data.

The data pipeline

Scroll sideways to see the whole diagram.

Pre-training data pipeline. Raw sources pass through extraction, language ID and cheap heuristic filters before the more expensive quality classifier. Deduplication, PII and safety filtering, and benchmark decontamination follow; the surviving documents are mixed by domain weights, tokenized and written as shuffled shards.

Extraction: turn HTML into clean text, drop boilerplate (navigation, cookie banners), keep structure where useful (headings, lists, code blocks, math).

Language identification: a fast classifier assigns a language and confidence; route documents to per-language pipelines.

Heuristic quality filters: rules on document length, mean word length, the ratio of symbols to words, the fraction of lines ending in ellipses, repeated lines or n-grams, and presence of common stopwords. Cheap, transparent, and they remove a lot of junk.

Model-based quality filtering: a classifier scores documents for quality: for example, trained to distinguish reference-quality text from random web pages, or trained on LLM-generated labels of educational value. Keeping the top fraction by score improves downstream benchmarks substantially per token. The risk is narrowing diversity, so thresholds are tuned with ablations.

Deduplication:

Exact: hash normalized documents or lines and drop repeats.
Near-duplicate: MinHash over n-gram shingles plus locality-sensitive hashing finds documents with high Jaccard similarity (e.g. ≥ 0.8) without comparing every pair.
Why it matters: duplicates waste compute, over-weight boilerplate, increase memorization (including of private data) and inflate benchmark scores through leakage.

Safety and privacy filtering: remove or mask PII patterns, filter toxic or illegal content, and respect opt-outs.

Decontamination: remove training documents that overlap with evaluation sets, typically by long n-gram matching (e.g. 13-grams), so benchmark results reflect generalization.

Mixtures and epochs

Mixture weights set how much of each source the model sees. They're chosen by small-scale ablations: train small models on candidate mixtures and compare evaluations, then scale up.
Repetition: when high-quality data is limited, repeating it for a few epochs is nearly as good as fresh data; returns diminish beyond roughly four epochs.
Annealing: many recipes end pre-training with a short phase that up-weights the highest-quality data (and long documents for context extension) while the learning rate decays. This measurably improves benchmarks.

Interview angles

Derive the loss and its gradient with respect to logits.
Explain perplexity vs. bits per byte, and why perplexity isn't comparable across tokenizers.
Describe a web-data pipeline end to end, including dedup and decontamination.
Discuss mixture selection with small-scale ablations, and repetition limits.