Research map / Learning theory and algorithms / Kernels and regression theory
Why Less is More (Sometimes): A Theory of Data Curation
Elvis Dohmatob, Mohammad Pezeshki, Reyhane Askari-Hemmat
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| curate the training set down to 1k hard examples instead of using the 114k OpenThinker set | Qwen2.5-32B · AIME 2024 (average) | Pass@1 (%): 50.2 -> 56.7 | Improved |
| curate 59k s1 pool down to 1k examples | Qwen2.5-32B · AIME 2024 (average) | Pass@1 (%): 53.3 -> 56.7 | Improved |
| scale training data from 1k to 1M examples | Qwen2.5-32B · AIME hard-level questions | Avg@8 (%): 28.4 -> 64.9 | Improved |
| scale from 10k examples to 114k OpenThinker | Qwen2.5-32B · AIME hard-level questions | Avg@8 (%): 52.1 -> 47.9 | Hurt |
| keep-hard pruning versus random pruning with abundant data and strong generator | synthetic linear model · Synthetic 2x2 regimes | test error | Improved |
| prune data (keep hard) with small n | synthetic linear model · Synthetic 2x2 regimes (small n) | test error | Hurt |
| prune data (keep hard) with a poor generator | synthetic linear model · Synthetic 2x2 regimes (poor generator, large n) | test error | Hurt |
| use keep-easy instead of keep-hard pruning when generator trained on only 160K examples | ViT-B/16 pseudo-labeling · ImageNet-1K val (generator n=160K) | top-1 accuracy | Improved |
| use keep-hard pruning when generator trained on 1.2M examples | ViT-B/16 pseudo-labeling · ImageNet-1K val (generator n=1.2M) | top-1 accuracy | Improved |
| apply keep-hard selection at each round of iterative retraining on pseudo-labels | ViT-B/16 iterative pseudo-labeling · ImageNet-1K val (iterative retraining) | top-1 accuracy | Improved |
Closest papers by meaning
- Scaling Laws for Data Filtering-- Data Curation cannot be Compute Agnostic (CVPR 2024 · similarity 0.86)
- The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track (NeurIPS 2024 · similarity 0.86)
- Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning (NeurIPS 2026 · similarity 0.85)
- Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes (ICML 2024 · similarity 0.85)
- Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.84)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
