Design the data pipeline for pre-training an LLM on web data.

30-second answer

Extract clean text from HTML, identify language, then apply cheap heuristic filters (length, symbol ratios, repetition, stopwords) and a model-based quality classifier. Deduplicate exactly with hashes and near-duplicates with MinHash LSH on n-gram shingles, since duplicates waste compute, increase memorization and leak benchmarks. Filter PII and toxic content, and decontaminate against evaluation sets with long n-gram matching. Choose mixture weights across web, code, math, books and multilingual data with small-model ablations, repeat scarce high-quality data only a few epochs, and anneal on the best data at the end.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

Ask about scale (trillions of tokens?), languages, sources, compute budget for processing, legal constraints (opt-outs, licences), and the target model size. Framing: "I'll design the pipeline stage by stage, explain why each matters with evidence, cover mixture selection and evaluation of data choices, and finish with risks."

5–15 min Core explanation: the pipeline

Scroll sideways to see the whole diagram.

Pre-training data pipeline. Raw sources pass through extraction, language ID and cheap heuristic filters before the more expensive quality classifier. Deduplication, PII and safety filtering, and benchmark decontamination follow; the surviving documents are mixed by domain weights, tokenized and written as shuffled shards.

Acquire: web crawl snapshots, code repositories (licence-filtered), books, papers, encyclopedias, math, curated multilingual sources; honour robots.txt and opt-outs.
Extract: HTML to text with boilerplate removal; keep headings, lists, tables, code and math.
Language ID: route by language; drop low-confidence documents.
Heuristic filters: document length, average word length, symbol-to-word ratio, repeated lines/paragraphs/n-grams, bullet-heavy or ellipsis-heavy pages, missing stopwords.
Model-based quality filter: score documents with a classifier and keep the top fraction.
Deduplicate: exact (hash) and near-duplicate (MinHash LSH).
Safety and privacy: PII detection and masking, toxicity and illegal-content filters.
Decontaminate: remove overlaps with evaluation sets.
Mix and tokenize: apply mixture weights, tokenize, shuffle, shard.

15–25 min Deep dive

Near-duplicate detection with MinHash LSH.

Represent each document by its set of word n-grams (e.g. 5-grams).
Compute k MinHash values (e.g. 128); the fraction of matching values estimates Jaccard similarity.
Split the signature into b bands of r rows; documents sharing any band hash become candidates. The probability two documents with similarity s become candidates is 1 − (1 − s^r)^b, an S-curve whose threshold you tune (e.g. ~0.8).
Cluster candidates and keep one per cluster.
Why: duplicates waste compute, encourage memorization (including of personal data), bias the model toward boilerplate and leak benchmark content. Deduplication has repeatedly been shown to improve model quality per token.

Quality classifiers.

Train a fast classifier (e.g. linear over n-gram features, or a small transformer) to separate high-quality reference text from random web text, or to predict LLM-assigned quality ratings (such as educational value).
Keep the top 10–30% by score, and validate with ablations: train small models on filtered vs. unfiltered data and compare benchmark curves.
Risks: homogenizing style, removing dialects or domains, and Goodharting toward the classifier.

Decontamination. Index long n-grams (e.g. 13-grams) of all evaluation sets you report on; drop or redact training documents with matches. Report contamination checks with results.

Mixture selection.

Train small proxy models on candidate mixtures and compare on a broad evaluation suite.
Up-sample high-value domains (code, math, reference text) within repetition limits (~4 epochs).
Use an annealing phase at the end with the highest-quality data.

25–32 min Trade-offs and risks

Decision	Trade-off
Aggressive quality filtering	Better benchmarks per token vs. less diversity and fewer tokens
Heavy dedup	Less memorization vs. losing legitimately repeated content (licences, common code)
More synthetic data	Fills gaps vs. model collapse risk and style homogenization
Strict PII filtering	Privacy vs. removing useful content (names in news, code with emails)

Track provenance and licences per document; you'll need them for opt-outs and audits.

32–37 min Production example

A team's first pipeline used only heuristics and exact dedup. Adding near-duplicate removal cut the token count by ~20% and, with the same compute, improved several benchmarks. Adding a model-based quality filter keeping the top ~30% of web text improved knowledge and reasoning benchmarks further, but hurt performance on informal-language tasks. They kept a small share of unfiltered web data in the mixture to restore diversity.

37–40 min Likely follow-ups

How do you evaluate a data change cheaply? Small proxy models (e.g. 1B parameters, tens of billions of tokens) with fixed recipes, compared on a broad evaluation suite with multiple seeds.
How do you handle code? Licence filtering, dedup at file and repository level, removing generated or minified files, and secret scanning.
How would you build a multilingual mixture? Per-language pipelines and quality models, temperature-based sampling to up-weight low-resource languages within repetition limits.

What interviewers listen for

A complete pipeline with reasons for each stage.
MinHash LSH explained concretely.
Model-based filtering validated by ablations.
Decontamination and mixture selection.

Red flags

"Crawl the web and train on it."
No mention of deduplication or contamination.