Research map / Large language models / Language, safety and interpretability

DataComp-LM: In search of the next generation of training sets for language models

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DCLM, a controlled testbed for language-model training-data curation with a 240T-token Common Crawl pool, fixed OpenLM training recipes at five scales, and 53 evaluations, and uses it to build DCLM-baseline via fastText model-based filtering.

Problem

Comparisons of training datasets for language models are confounded by differing architectures, compute, and hyperparameters, and recent open-weight models do not disclose their training data, so it is unclear which data curation strategies work best.

Approach

A standardized benchmark where participants curate data from a fixed raw pool (DCLM-Pool) and are scored by training models with a fixed recipe at multiple compute scales; the baseline finds that a fastText classifier trained on instruction-formatted (OpenHermes 2.5 + ELI5) positives and RefinedWeb negatives, keeping the top 10%, gives the best data.

Key results

• DCLM-baseline 7B trained on 2.6T tokens (with StarCoder and ProofPile2) reaches 63.7% MMLU 5-shot, 57.1 CORE, versus MAP-Neo 57.1 MMLU at 4.5T tokens (open data), and comparable to Llama 3 8B (66.2 MMLU).
• Ranking of curation methods at smaller scales correlates with 7B-1x results (Pearson r=0.838, 0.956, 0.982 for 400M, 1B, 3B).
• fastText filter with OpenHermes 2.5 + ELI5 positives and top-10% threshold gives the best 1B-scale CORE (30.2) among filters tested, and 41.0 CORE at 7B-1x.

Measured effects reported in the paper

Surprising findings

• Mixing high-quality sources (Wikipedia, books, StackExchange, arXiv, GitHub) hurts performance on DCLM-baseline, though it helps weaker Common Crawl subsets.
• Human quality judgments have only limited value in identifying high-quality training data, and a simple bigram classifier performs best among the classifiers tried.

Limitations

• Experiments were run once per setting without error bars due to compute cost, and most ablations were done individually.
• Models were limited to 7B parameters and mainly one tokenizer (GPT-NeoX), so larger-scale and multilingual behavior is untested.
• Models trained on DCLM-baseline are weaker on code and math than on language understanding tasks.

Experimental setup

Datasets: DCLM-Pool DCLM-baseline C4 RefinedWeb RedPajama Dolma-V1 OpenHermes 2.5 ELI5 StarCoder ProofPile2 Wikipedia OpenWebText2

Benchmarks: MMLU 5-shot CORE (22 tasks) EXTENDED (53 tasks) HellaSwag ARC-E AlpacaEval 2.0

Metrics: accuracy centered accuracy win-rate (LC) Pearson correlation

Baselines: C4 RefinedWeb RedPajama Dolma-V1.7 FineWeb-Edu Llama 3 8B Mistral-7B-v0.3 MAP-Neo OLMo-1.7 Gemma-7B Llama 2 7B DeepSeek 7B

Builds on: OpenLM RefinedWeb DataComp LLM-Foundry Chinchilla scaling laws

Closest papers by meaning

• Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.91)
• RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.90)
• Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.90)
• Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.89)
• DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.89)
• Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.89)