Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
DataRater: Meta-Learned Dataset Curation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Meta-learns a small non-causal transformer "DataRater" via meta-gradients (MixFlow-MG) that assigns per-sample weights to pretraining data, enabling top-K batch filtering that cuts training compute for matching performance on low-quality corpora.
Problem
Pretraining data curation is largely manual (hand-crafted heuristics and tuned mixtures); the paper seeks an automated, scalable way to learn which data points are valuable for training efficiency on held-out data.
Approach
Parameterise per-sample value with a learned DataRater scoring function, softmax-normalised within each batch to weight the inner-loop gradient, and optimise it end-to-end with meta-gradients (back-propagating through unrolled inner training) so that the inner models learn faster on held-out data from the same distribution.
Key results
- Filtering C4/noclean and the Pile with the DataRater yields up to 46.6% net compute gain for 1B models, with improved final performance on lower-quality datasets.
- DataRater trained on 400M inner models generalises across scales (50M to 1B): 73 of 84 metric comparisons showed improvements.
- Versus perplexity-based filtering at 150M scale, DataRater achieved best performance in 16 of 21 evaluations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Filter training data using DataRater | 1B language models trained on the Pile and C4/noclean · Pile and C4/noclean | compute needed to match baseline performance: up to 46.6% net compute gain | Helped |
| Filter training data using DataRater | Language models across the evaluated datasets and scales · 3 input datasets, 4 model scales, 7 metrics | performance: 73 / 84 showed improvements | Helped |
| Filter data using DataRater rather than the best perplexity-based variant, PPLX (Mid) | 150M language models · Pile, C4/noclean, C4 | performance across evaluation metrics: 16 out of 21 evaluations | Mixed |
| Use eight 400M inner models instead of 150M inner models (ablation) | DataRater meta-training · C4/noclean | relative performance improvement | Helped |
| Use a 50% discard fraction (ablation) | DataRater trained with different inner-model scales · C4/noclean | relative performance improvement | No clear effect |
| Use two inner model updates for the meta-gradient unroll instead of one, four, or eight (ablation) | DataRater · Meta-learning | DataRater performance: 1, 2, 4 and 8 inner model updates | No clear effect |
| Use sequence-level filtering instead of token-level weighting or coarse-mixture weighting (ablation) | DataRater · Training-efficiency meta-objective | performance | Helped |
| Use PPLX (Mid) filtering instead of DataRater filtering (ablation) | 150M language models · Pile, C4/noclean, C4 | performance across evaluation metrics: four evaluations | Mixed |
| Use PPLX (Mid) filtering instead of no filtering (ablation) | 150M language models · C4 | validation set performance: 90% | Helped |
| Increase the discard fraction according to the dataset-specific selected proportion (ablation) | DataRater-filtered training · C4: 10%; C4/noclean: 50%; Pile: 75% | validation NLL: 10%, 50% and 75% | Helped |
| Meta-train DataRater with eight 400M inner models rather than other tested inner-model scales (ablation) | DataRater · C4/noclean | relative performance improvement: 8 x 400M | Helped |
| Use perplexity filtering rather than no filtering (ablation) | PPLX (Mid) · C4 | SIQA performance | Hurt |
Surprising findings
- Gains are small or trade off on the higher-quality C4 dataset, with compute-matching undefined in some C4 cases.
- Learned DataRater score correlates only weakly (0.20-0.22) with the composite C4 quality filter, suggesting it captures value not reflected in standard heuristics.
Limitations
- Effectiveness depends on the choice of meta-objective and held-out data distribution; only same-distribution objective evaluated in main experiments.
- Meta-gradient computation remains expensive for larger models with dense inner updates; scalability to very large foundation models is an open question.
- Gains are minimal on the already high-quality C4 dataset.
Experimental setup
Datasets: The Pile C4/noclean C4 (en) Wikipedia (en) HellaSwag SIQA PIQA ARC Easy CommonsenseQA
Benchmarks: The Pile validation NLL C4/noclean validation NLL C4 validation NLL Wikipedia NLL HellaSwag SIQA PIQA ARC Easy CommonsenseQA
Metrics: negative log likelihood accuracy fraction of compute to match baseline (FLOPs)
Baselines: PPLX (Top) PPLX (Mid) PPLX (Bot) no filtering
Builds on: MixFlow-MG Chinchilla RHO-LOSS Podracer
Closest papers by meaning
- Scaling Laws for Data Filtering-- Data Curation cannot be Compute Agnostic (CVPR 2024 · similarity 0.86)
- Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws (ICLR 2025 · similarity 0.85)
- The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track (NeurIPS 2024 · similarity 0.85)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.84)
- MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters (ICML 2025 · similarity 0.84)
- Improving Generalization via Meta-Learning on Hard Samples (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.