Research map / Large language models / AI and society
BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a 46-criterion assessment framework for AI benchmark quality, scores 24 foundation-model and non-foundation-model benchmarks against it, and releases a checklist and living repository of assessments.
Problem
AI benchmarks vary widely in quality (design, implementation, documentation, maintenance), yet no structured assessment of benchmark quality existed, and many benchmarks lack statistical significance reporting and replicability.
Approach
A lifecycle-based assessment of 46 criteria derived from stakeholder interviews and literature from other fields, scored on a 0/5/10/15 scale into design and usability scores, plus a developer checklist and a continuously updated public website of benchmark assessments.
Key results
- MMLU scored lowest in the assessment (weighted average usability 5.5) while GPQA scored 11.0.
- Benchmarks averaged 3.75 on including a script to replicate results and 5.62 on reporting statistical significance.
- 14 of 24 benchmarks did not perform multiple evaluations or report significance or uncertainty; 17 of 24 lacked scripts to replicate initial results.
Surprising findings
- Implementation and maintenance stages are the weakest areas across FM and non-FM benchmarks.
- Only 3 of 24 benchmarks implemented a build status in their repository.
Limitations
- Equal weighting of all criteria and coarse 4-level scoring may miss nuances and allow gaming by fulfilling easy criteria.
- Construct validity of benchmarks is not assessed, requiring domain-expert analysis beyond scope.
- Framework covers only public benchmarks, not private ones.
Experimental setup
Datasets: AgentBench ARC Challenge BBQ BIG-bench BOLD Codex HumanEval DecodingTrust GPQA GSM8k HellaSwag Machiavelli MLCommons AI Safety v0.5
Benchmarks: AgentBench ARC Challenge BBQ BIG-bench BOLD Codex HumanEval DecodingTrust GPQA GSM8k HellaSwag Machiavelli MLCommons AI Safety v0.5
Metrics: design score usability score lifecycle stage subscores criteria score (0/5/10/15)
Builds on: data cards FAIR principles NeurIPS checklist
Closest papers by meaning
- Establishing Best Practices in Building Rigorous Agentic Benchmarks (NeurIPS 2025 · similarity 0.90)
- Fantastic Bugs and Where to Find Them in AI Benchmarks (NeurIPS 2025 · similarity 0.90)
- PaperBench: Evaluating AI’s Ability to Replicate AI Research (ICML 2025 · similarity 0.88)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.87)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.87)
- RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.