Research map / Learning theory and algorithms / Prediction and decision losses
Your Benchmark Is an Empirical Measure Over Difficulty
Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes viewing a benchmark as an empirical measure over item difficulty, formalizing measure mismatch and coverage gaps as failure modes of benchmark evaluation.
Problem
Benchmarks may not reflect the intended difficulty distribution of the target capability, and may leave difficulty regions uncovered; the paper aims to formalize these failures.
Approach
Treats a benchmark as inducing an empirical measure over difficulty and uses this measure-based view to formally define two failure modes: measure mismatch (benchmark difficulty distribution differs from intended one) and coverage gaps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add BenchFill difficulty-targeted rewriting (3 inserted items) to benchmarks | benchmarks (8 tested) · 8 benchmarks | Top-3 gap sum reduction (%): 21.09% - 38.88% reduction; average 32.67 | Improved |
| use BenchFill adaptive memory-guided search instead of budget-matched uniform-parent rewriting (ablation) | gap-filling rewriter · 8 benchmarks average | Top-3 gap sum reduction (%): 17.52 -> 32.67 | Improved |
| use BenchFill instead of uniform-parent baseline for hitting target difficulty (ablation) | gap-filling rewriter · 8 benchmarks average | Target deviation: 0.048 -> 0.005 | Improved |
| compare BenchFill with oracle midpoint insertion | BenchFill · AIME24/25/26 | Top-3 gap sum reduction (%): 35.57 vs 35.57 on AIME24 | No clear effect |
| Gap-fill AIME24/AIME25/AIME26/AMC23 with BenchFill to expand items | math benchmarks · AIME24, AIME25, AIME26, AMC23 | largest coverage gap: 30, 30, 30, 40 -> 50, 52, 53, 80 items | Improved |
| benchmarks differ from uniform difficulty target | 25 benchmarks · 25 benchmarks | DDM >= 0.10: 10 of 25 benchmarks | Hurt |
| small benchmark size (e.g. 30 items AIME) increases coverage gaps | AIME24, AIME25 · AIME | Top-5 gap sum: about 0.5 | Hurt |
Closest papers by meaning
- Fantastic Bugs and Where to Find Them in AI Benchmarks (NeurIPS 2025 · similarity 0.85)
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark (ICLR 2025 · similarity 0.84)
- Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs (NeurIPS 2026 · similarity 0.84)
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.83)
- Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models (NeurIPS 2026 · similarity 0.83)
- How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
