AI Paper Atlas

Research map / Learning theory and algorithms / Prediction and decision losses

Your Benchmark Is an Empirical Measure Over Difficulty

Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang

NeurIPS 2026 · Learning theory and algorithms › Prediction and decision losses · approach: Adaptive testing and experiments · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes viewing a benchmark as an empirical measure over item difficulty, formalizing measure mismatch and coverage gaps as failure modes of benchmark evaluation.

Problem

Benchmarks may not reflect the intended difficulty distribution of the target capability, and may leave difficulty regions uncovered; the paper aims to formalize these failures.

Approach

Treats a benchmark as inducing an empirical measure over difficulty and uses this measure-based view to formally define two failure modes: measure mismatch (benchmark difficulty distribution differs from intended one) and coverage gaps.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add BenchFill difficulty-targeted rewriting (3 inserted items) to benchmarksbenchmarks (8 tested) · 8 benchmarksTop-3 gap sum reduction (%): 21.09% - 38.88% reduction; average 32.67Improved
use BenchFill adaptive memory-guided search instead of budget-matched uniform-parent rewriting (ablation)gap-filling rewriter · 8 benchmarks averageTop-3 gap sum reduction (%): 17.52 -> 32.67Improved
use BenchFill instead of uniform-parent baseline for hitting target difficulty (ablation)gap-filling rewriter · 8 benchmarks averageTarget deviation: 0.048 -> 0.005Improved
compare BenchFill with oracle midpoint insertionBenchFill · AIME24/25/26Top-3 gap sum reduction (%): 35.57 vs 35.57 on AIME24No clear effect
Gap-fill AIME24/AIME25/AIME26/AMC23 with BenchFill to expand itemsmath benchmarks · AIME24, AIME25, AIME26, AMC23largest coverage gap: 30, 30, 30, 40 -> 50, 52, 53, 80 itemsImproved
benchmarks differ from uniform difficulty target25 benchmarks · 25 benchmarksDDM >= 0.10: 10 of 25 benchmarksHurt
small benchmark size (e.g. 30 items AIME) increases coverage gapsAIME24, AIME25 · AIMETop-5 gap sum: about 0.5Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy