Research map / Large language models / Language, safety and interpretability
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FineWeb, a 15T-token English web pretraining dataset from 96 Common Crawl snapshots, plus FineWeb-Edu (1.3T tokens) filtered by a Llama-3-annotated educational classifier, with ablations documenting each curation choice.
Problem
Open LLM pretraining datasets are poorly documented and often lag proprietary ones; the design choices (text extraction, filtering, deduplication) that most affect downstream performance are not publicly understood.
Approach
Builds a large Common Crawl dataset through systematic data-ablation experiments: trafilatura extraction from WARC, per-snapshot MinHash deduplication (rather than global), C4 and newly derived heuristic filters, and an educational-quality classifier trained on Llama-3-70B annotations to create FineWeb-Edu.
Key results
- FineWeb-Edu ablation (1.71B model, 350B tokens) raises MMLU from 33% to 37% and ARC from 46% to 57% versus FineWeb.
- FineWeb-Edu reaches 33.6% MMLU at 38B tokens, while Matrix, the second-best dataset, needs about 300B tokens for similar accuracy.
- Individual-snapshot MinHash deduplication yields 20T tokens and matches RefinedWeb performance, whereas global deduplication underperforms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| filter FineWeb for educational content with a minimum score threshold of 3 | FineWeb · MMLU, 350B tokens | accuracy: 33% -> 37% | Helped |
| filter FineWeb for educational content with a minimum score threshold of 3 | FineWeb · ARC, 350B tokens | score: 46% -> 57% | Helped |
| extract text from WARC with Trafilatura instead of using WET text (ablation) | Common Crawl training data · downstream benchmark suite | performance | Helped |
| apply base URL, language, quality, and repetition filtering (ablation) | WARC-extracted Common Crawl data · downstream benchmark suite | performance | Helped |
| apply global MinHash deduplication across snapshots instead of no deduplication (ablation) | base-filtered FineWeb data · aggregate task score | aggregate score | No clear effect |
| deduplicate each Common Crawl snapshot independently with MinHash (ablation) | base-filtered FineWeb data · average benchmark score | average score | Helped |
| apply URL deduplication after individual MinHash deduplication (ablation) | FineWeb data · downstream benchmark suite | performance | Hurt |
| apply line deduplication after individual MinHash deduplication (ablation) | FineWeb data · downstream benchmark suite | performance | Hurt |
| apply C4's terminal punctuation filter (ablation) | base-filtered, individually deduplicated 2019-18 crawl · HellaSwag | performance | Helped |
| apply the selected C4 filters except the terminal punctuation filter (ablation) | base-filtered, individually deduplicated 2019-18 crawl · HellaSwag | performance | Helped |
| apply FineWeb's three custom heuristic filters together (ablation) | base-filtered, individually deduplicated data with C4 filters · aggregate benchmark score | aggregate score: about 1% | Helped |
| use the FineWeb-Edu educational filter instead of unfiltered FineWeb (ablation) | FineWeb · MMLU, 350B tokens | accuracy: approximately 12% | Helped |
Surprising findings
- Global MinHash deduplication across all 96 snapshots removed up to 90% of data in older crawls and gave only modest gains, with the kept data of worse quality than the removed data.
- The C4 terminal punctuation filter gave the largest single HellaSwag gain but removed around 30% of tokens, so it was excluded.
Limitations
- Almost all data is web text from Common Crawl, with no books, code, or other sources.
- Most ablations ran at small scale (1.71B params, up to 350B tokens) due to compute limits.
- Evaluation is limited to academic benchmarks without instruction tuning or alignment.
Experimental setup
Datasets: Common Crawl (96 snapshots) FineWeb RefinedWeb C4 Dolma The Pile SlimPajama RedPajama2 Matrix CC-100 Colossal-OSCAR Paloma
Benchmarks: CommonSenseQA HellaSwag OpenBookQA PIQA SIQA WinoGrande ARC MMLU Paloma
Metrics: accuracy aggregate accuracy perplexity F1
Baselines: RefinedWeb C4 Dolma The Pile SlimPajama RedPajama2 Matrix CC-100 Colossal-OSCAR
Builds on: RefinedWeb C4 MassiveText datatrove
Closest papers by meaning
- Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs (NeurIPS 2024 · similarity 0.89)
- SWEb: A Large Web Dataset for the Scandinavian Languages (ICLR 2025 · similarity 0.89)
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.88)
- Organize the Web: Constructing Domains Enhances Pre-Training Data Curation (ICML 2025 · similarity 0.88)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.88)
- The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.