Research map / Label-efficient and robust learning / Recognition and long-tail
DsDm: Model-Aware Dataset Selection with Datamodels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames LM dataset selection as optimizing trained-model target loss and approximates it with linear datamodels (TRAK) to select C4 subsets (DsDm), yielding ~2x compute multiplier over random selection.
Problem
Standard similarity-based data selection (toward "high quality" reference sources) for LM pretraining often fails to beat random selection, and finding the subset that best reduces target-task loss is combinatorially infeasible.
Approach
Approximate the trained-model target loss as a linear function of the training subset's indicator vector using datamodels estimated efficiently with TRAK, then select the size-k subset minimizing this estimate (top-k of averaged datamodel weights over target samples), so selection models how learning uses data rather than textual similarity.
Key results
- DsDm-selected data yields 1.3B models that match models trained with 2x the compute on random C4 data (1.8B Chinchilla-optimal) on mean benchmark accuracy.
- Classifier and DSIR selections do not consistently outperform random selection across compute budgets; e.g., on SQuAD and CS-Algorithms they underperform random.
- DsDm improves target-task log-probability across all four tasks, and 760M/1.3B DsDm-selected models outperform a 10x compute random-data model on target tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select training data with DsDm using LAMBADA, SQuAD, and Jeopardy as joint target tasks instead of random selection | GPT-2 language models · Held-out benchmarks, 1.3B model | accuracy (%): 63.0 vs 62.0; 31.2 vs 33.4; 69.0 vs 68.9; 88.2 vs 86.4; 42.3 vs 44.9; 51.1 vs 52.2; 58.0 vs 54.9; 25.5 vs 18.8; 15.6 vs | Mixed |
| Use DsDm rather than textual-similarity selection with DSIR or Classifier | GPT-2 language models · Held-out benchmarks across compute budgets | mean benchmark performance: 2× | Helped |
| Target a single task rather than the combined LAMBADA, SQuAD, and Jeopardy tasks | DsDm · Held-out benchmark categories | accuracy | Mixed |
Surprising findings
- Selecting data similar to high-quality sources (Wikipedia, Books1, OpenWebText) via Classifier or DSIR often fails to beat random selection, and can hurt performance.
- DsDm's top-selected examples often contain seemingly nonsense text yet yield the best models on some tasks.
Limitations
- Linear datamodels and TRAK approximation are approximate to the true trained-model loss
- Targets must be chosen by the practitioner; unrelated benchmark categories can be hurt by targeting (e.g., LAMBADA reduces world knowledge)
- Evaluated only up to 1.8B-parameter GPT-2 style models trained on C4
Experimental setup
Datasets: C4 en.noblocklist SQuAD Jeopardy LAMBADA CS-Algorithms (BIG-bench) Wikipedia Books1 OpenWebText
Benchmarks: LAMBADA CS-Algorithms SQuAD Jeopardy copa openbook_qa piqa cbt hellaswag winogrande boolq coqa
Metrics: mean log-probability of label accuracy compute multiplier
Baselines: Random DSIR Classifier SemDeDup
Builds on: datamodels TRAK DSIR Classifier data selection (GPT-3) SemDeDup
Closest papers by meaning
- Data Selection via Optimal Control for Language Models (ICLR 2025 · similarity 0.88)
- The Best of Both Worlds: Bridging Quality and Diversity in Data Selection with Bipartite Graph (ICML 2025 · similarity 0.88)
- GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration (NeurIPS 2024 · similarity 0.88)
- A CLIP-Powered Framework for Robust and Generalizable Data Selection (ICLR 2025 · similarity 0.87)
- Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025 · similarity 0.87)
- Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.