Research map / Large language models / AI and society
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, Vilém Zouhar, Srishti Yadav, Chenxi Whitehouse, Dayeon Ki, Jennifer Mickel, Leshem Choshen and 25 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase benchmark age (past 24 months to over 60 months) | LLM benchmarks · 60 text-based LLM benchmarks | proportion saturated: 42.9% -> 54.5% | Hurt |
| use a private held-out test set instead of public test data | LLM benchmarks · 60 benchmarks (56 public, 4 private) | S_index | No clear effect |
| use open-ended instead of closed-ended output format | LLM benchmarks · 60 benchmarks (28 closed, 31 open) | S_index: age-balanced, p=0.40 | No clear effect |
| use multilingual instead of English-only benchmarks | LLM benchmarks · 60 benchmarks (16 multilingual, 44 English-only) | saturation rate: mean age 32.9 vs 48.9 months | No clear effect |
| use non-templated instead of templated benchmarks | LLM benchmarks · 60 benchmarks (46 non-templated, 14 templated) | saturation behaviour: p=0.10 | No clear effect |
| use expert-curated instead of crowdsourced benchmarks | LLM benchmarks · 60 benchmarks | saturation | Improved |
| larger test sets | LLM benchmarks · 60 benchmarks, joint regression | S_index | Improved |
| higher citation counts, controlling for age | LLM benchmarks · 60 benchmarks | S_index: rho = 0.22, p = 0.12 | No clear effect |
| smaller top-k (k=3) in saturation index (ablation) | Saturation index · sensitivity analysis | variance of estimate | Hurt |
| set alpha=1 in effective test size (ablation) | Saturation index · sensitivity analysis | dependence on test set size | Hurt |
Closest papers by meaning
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.90)
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026 · similarity 0.88)
- Minerva: A Programmable Memory Test Benchmark for Language Models (ICML 2025 · similarity 0.87)
- Prescriptive Scaling Reveals the Evolution of Language Model Capabilities (ICML 2026 · similarity 0.87)
- PACE: A Proxy for Agentic Capability Evaluation (NeurIPS 2026 · similarity 0.87)
- The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
