Research map / Large language models / Language, safety and interpretability
Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds 24-category topic and format taxonomies for CommonCrawl, distills Llama-3.1-405B annotations into 140M classifiers, and uses RegMix-optimized domain mixtures (alone or with quality filters) to improve 1B-model downstream accuracy.
Problem
Web-crawled pre-training corpora are unstructured, making their contents hard to inspect and making systematic data curation (especially domain balancing) difficult.
Approach
Organize web pages into two complementary domain taxonomies (topic and format) annotated by a distilled small classifier, then optimize domain mixtures for downstream tasks with an adapted RegMix pipeline and combine them with quality filters.
Key results
- Topic x Format domain mixing raises the 1B model's 9-task average from 51.6 (baseline) to 54.6, a 3.0-point gain.
- Adding Topic x Format mixing to FineWeb-Edu lifts its average from 54.2 to 56.2 and its HellaSwag score from 56.0 to 62.5.
- Implicit Topic x Format mixture of FineWeb-Edu recovers 84% of its average-task gain from mixing alone, while DCLM-fasttext is only approximated to 35%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reweight topic domains, format domains, or k-means clusters | Baseline corpus · OLMES 5-task suite average | accuracy: 51.6 -> 53.2; 51.6 -> 53.7; 51.6 -> 53.4 | Helped |
| Combine topic and format mixtures | Baseline corpus · OLMES 5-task suite average | accuracy: 51.6 -> 54.6 | Helped |
| Add topic × format domain mixing | FineWeb-Edu · OLMES 5-task suite average | accuracy: 54.2 -> 56.2 | Helped |
| Add topic × format domain mixing | DCLM-fasttext · OLMES 5-task suite average | accuracy: 55.1 -> 56.1 | Helped |
| Add topic × format domain mixing | FineWeb-Edu · MMLU | accuracy: 34.3 -> 34.2 | Hurt |
| Add topic × format domain mixing | FineWeb-Edu · HellaSwag | accuracy: 56.0 -> 62.5 | Helped |
| Remove 2-stage training from the topic classifier (ablation) | WebOrganizer topic classifier · Topic classifier validation | average accuracy: 91.8 -> 93.5 | Helped |
| Remove 2-stage training from the topic classifier (ablation) | WebOrganizer topic classifier · Topic classifier validation | worst-group accuracy: 84.3 -> 87.1 | Helped |
| Remove 2-stage training from the format classifier (ablation) | WebOrganizer format classifier · Format classifier validation | worst-group accuracy: 74.1 -> 80.5 | Helped |
| Remove URL features from the format classifier (ablation) | WebOrganizer format classifier · Format classifier validation | average accuracy: 88.9 -> 91.8 | Helped |
| Remove URL features from the topic classifier (ablation) | WebOrganizer topic classifier · Topic classifier validation | average accuracy: 92.1 -> 93.5 | Helped |
Surprising findings
- Quality filters' implicit domain mixtures explain much of FineWeb-Edu's gain, but DCLM-fasttext's gains depend much more on selecting specific documents within domains.
- Domain-level reweighting of the corpus does not significantly reduce perplexity relative to baseline, whereas quality filtering raises perplexity substantially.
Limitations
- Predicted mixtures are based on small (50M) model runs and may not transfer across model scales.
- Taxonomies are one of many valid choices and are single-label, which can miss multi-topic documents.
- Classifiers have a gap to the 405B teacher (about 4-5% error on confident annotations).
Experimental setup
Datasets: DCLM RefinedWeb-based CommonCrawl pool (200B token corpus) DataComps-LM 1b-1x pool
Benchmarks: MMLU HellaSwag PIQA WinoGrande CommonSenseQA Social IQa ARC-easy ARC-challenge OpenBookQA
Metrics: accuracy (5-shot, OLMES) perplexity Spearman correlation classifier accuracy
Baselines: Random sampling baseline corpus k-means clusters FineWeb-Edu DCLM-fasttext
Builds on: RegMix DataComps-LM FineWeb-Edu DCLM-fasttext OLMES
Closest papers by meaning
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (NeurIPS 2024 · similarity 0.88)
- Nemotron-CLIMB: Clustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (NeurIPS 2025 · similarity 0.86)
- DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.86)
- SWEb: A Large Web Dataset for the Scandinavian Languages (ICLR 2025 · similarity 0.86)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.85)
- Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.