Research map / Label-efficient and robust learning / Recognition and long-tail
A Meta-Analysis of Overfitting in Machine Learning
Rebecca Roelofs, Vaishaal Shankar, Benjamin Recht, Sara Fridovich-Keil, Moritz Hardt, John Miller, Ludwig Schmidt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| repeated test set reuse (adaptive submissions) on public leaderboard | Kaggle accuracy competitions · 34 accuracy competitions | public-private accuracy gap | No clear effect |
| restricting to top 10% of submissions | overfitting analysis · Competitions 3788, 7634 | public-private accuracy gap: about 1% | Mixed |
| small or non-i.i.d. test sets | Kaggle competitions · outlier competitions | mean accuracy difference | Hurt |
Closest papers by meaning
- What Fits (Into Few Tokens) Doesn't Overfit: Compression and Generalization in ML Research Agents (NeurIPS 2026 · similarity 0.87)
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.86)
- A hierarchical decomposition for explaining ML performance discrepancies (NeurIPS 2024 · similarity 0.85)
- A Survey of Learning Criteria Going beyond the Usual Risk (Abstract Reprint) (AAAI 2024 · similarity 0.85)
- Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining (NeurIPS 2025 · similarity 0.85)
- What makes unlearning hard and what to do about it (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
