Research map / Large language models / Reasoning and chain of thought
Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that repeating aggressively filtered DCLM-baseline for up to 10 epochs (with scaled weight decay) can beat one epoch of the 10x larger RefinedWeb superset, and proposes document-level count manipulation guided by a quality score to build better subsampled datasets.
Problem
As LLM compute budgets grow, heavily filtered and deduplicated datasets provide too few unique tokens, raising the question of whether repeating aggressively filtered data can outperform larger, lower-quality datasets, and how document-level duplicate counts relate to data quality.
Approach
Repetition of a higher-quality filtered dataset can beat a single epoch on a larger lower-quality superset if weight decay is scaled with repeat count; document duplicate counts are correlated with quality, so manipulating per-document counts (greedy or linear functions over a combined quality/duplicate rank) yields better subsets than uniform subsampling or global deduplication.
Key results
- Ten epochs of DCLM-baseline with 2x weight decay reaches 0.523 CORE vs 0.550 for one epoch of RefinedWeb at 12.6B params/252B tokens (WD 1x for repeats is 0.489), so weight decay tuning is needed for repeats to help.
- For 7B models at 138B tokens, Greedy 1 copy (ensemble of FastText and duplicate rank) reaches 46.1 CORE and 46.0 MMLU vs 44.4 / 42.2 for uniform subsampling.
- Duplicate-aware subsampling performs worst (34.1 CORE), showing that keeping many duplicates hurts performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase weight decay from 1x to 2x (ablation) | DCLM-baseline training · 12.6B parameters, 252B tokens, 10 epochs | centered core metric: 0.489 -> 0.523 | Helped |
| Increase weight decay from 1x to 2x (ablation) | DCLM-baseline training · 12.6B parameters, 252B tokens, 5 epochs | centered core metric: 0.532 -> 0.539 | Helped |
| Increase weight decay from 1x to 2x (ablation) | DCLM-baseline training · 12.6B parameters, 252B tokens, 1 epoch | centered core metric: 0.550 -> 0.547 | Hurt |
| Use duplicate-aware subsampling instead of uniform subsampling | DCLM-baseline, 7B, 138B tokens · DCLM-baseline | CORE (22 task average): 44.4 -> 34.1 | Hurt |
| Select documents by ensemble ranking and keep one copy each | DCLM-baseline count manipulation · 7B, 138B tokens | CORE (22 task average): 44.4 -> 46.1 | Helped |
| Select documents by ensemble ranking and keep one copy each | DCLM-baseline count manipulation · 7B, 138B tokens | MMLU: 42.2 -> 46.0 | Helped |
| Select documents by duplicate count instead of ensemble ranking (ablation) | DCLM-baseline count manipulation · 7B, 138B tokens | MMLU: 46.0 -> 28.5 | Hurt |
| Use linear count assignment up to 5 copies instead of greedy diversity (ablation) | DCLM-baseline duplicate-aware subsample · 7B, 138B tokens | CORE (22 task average): 40.4 -> 41.2 | Helped |
| Use linear count assignment up to 5 copies instead of greedy diversity (ablation) | DCLM-baseline duplicate-aware subsample · 7B, 138B tokens | MMLU: 28.3 -> 30.4 | Helped |
| Use DCLM-baseline for the final 10% or 20% of training instead of shuffling the same proportion with RefinedWeb (ablation) | 7B continual pre-training · 138B tokens | Core: 41.3 -> 41.0; 41.6 -> 41.6 | No clear effect |
Surprising findings
- Global deduplication before subsampling performs much worse (41.7 CORE) than uniform subsampling of DCLM-baseline, suggesting high duplicate-count documents tend to be higher quality.
- Repeating DCLM-baseline with too few unique tokens can perform worse than one epoch of the larger RefinedWeb superset.
Limitations
- Experiments are smaller scale than frontier models, so trends may not hold at frontier scale.
- Focused only on web-scraped general-language datasets; code and math may differ.
- No error bars due to cost of large-scale training runs.
Experimental setup
Datasets: DCLM-baseline RefinedWeb (DCLM v2) C4 Zyda-2 DCLM-dedup
Benchmarks: DCLM CORE (22 tasks) MMLU C4 validation perplexity
Metrics: centered core accuracy MMLU few-shot accuracy perplexity
Baselines: RefinedWeb (DCLM reproduction) C4 Zyda-2 DCLM-dedup uniform subsampling global deduplication duplicate-aware subsampling
Builds on: DCLM Muennighoff et al. data-constrained scaling Chinchilla scaling laws
Closest papers by meaning
- A Bitter Lesson for Data Filtering (NeurIPS 2026 · similarity 0.89)
- Scaling Laws for Data Filtering-- Data Curation cannot be Compute Agnostic (CVPR 2024 · similarity 0.89)
- Bridging Compute- and Data-Optimal Pretraining (NeurIPS 2026 · similarity 0.88)
- DataComp-LM: In search of the next generation of training sets for language models (NeurIPS 2024 · similarity 0.87)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.87)
- Data Filtering Networks (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.