Research map / Large language models / AI and society
DataPerf: Benchmarks for Data-Centric AI Development
Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William A Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk, Jessica Quaye, Charvi Rastogi and 35 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Cleanlab outlier rejection instead of nested cross-fold selection | Selection for Speech baselines · English, 25 samples | macro F1: 0.32 -> 0.49 | Improved |
| use Cleanlab instead of nested cross-fold selection | Selection for Speech baselines · Indonesian, 25 samples | macro F1: 0.36 -> 0.37 | No clear effect |
| use DataScope Shapley approximation instead of random ordering for label cleaning | Debugging for Vision · Debugging for Vision | % of samples fixed to reach threshold: 51.75 -> 15.54 | Improved |
| buy uniformly from all sellers instead of from a single random seller | Data Acquisition baselines · market instance 4 | accuracy: 0.679 -> 0.742 | Improved |
| dataset iteration (selection, augmentation) with fixed model | DCAI Roman numeral challenge · DCAI | accuracy: 64.4% -> 85.8% | Improved |
Closest papers by meaning
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.87)
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (ICLR 2025 · similarity 0.86)
- The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization (NeurIPS 2025 · similarity 0.85)
- Benchmark Data Repositories for Better Benchmarking (NeurIPS 2024 · similarity 0.85)
- MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges? (NeurIPS 2025 · similarity 0.85)
- MixEval-X: Any-to-any Evaluations from Real-world Data Mixture (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
