Research map / Large language models / AI and society

Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs

NeurIPS 2026 · Large language models › AI and society · approach: Critiques of LLM claims and judging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Benchmark Health Index (BHI), a data-driven framework scoring 106 LLM benchmarks from 2025 technical reports on capability discrimination, anti-saturation, and impact, combined via CRITIC weighting.

Problem

LLM benchmarks are becoming unreliable due to score inflation, saturation, contamination, and selective reporting, and there is no quantitative macro-level framework for auditing benchmark health and selecting benchmarks.

Approach

A three-axis index (discrimination via EDR and RCV, anti-saturation via capability-weighted static resistance and dynamic trend projection, and impact via industry adoption and community heat) calibrated with leave-one-benchmark-out model capability estimates and fused with CRITIC objective weights.

Key results

• Humanity's Last Exam ranks first with BHI 0.6686, followed by SimpleQA (0.6264) and ZeroBench (0.5877).
• Under model dropout at eta=0.15, BHI-CRITIC Spearman sigma is 0.0052 vs 0.0150 for equal weighting.
• At noise sigma=0.20, BHI-CRITIC Spearman correlation is 0.9682 vs 0.9610 for BHI-EW.

Surprising findings

• AIME 2025 has high impact (S_Imp 0.8065) but very low anti-saturation (S_AS 0.1252), indicating popularity decoupled from remaining headroom.
• C-Eval and GSM8K show very low anti-saturation scores, suggesting contamination or saturation in widely used subject and math benchmarks.

Limitations

• Original technical reports often lack implementation details, adding residual variance to normalized scores.
• Aligning unbounded metrics may compress extreme capability ranges, yielding conservative anti-saturation estimates.
• No cross-version imputation within benchmark families causes localized sparsity.

Experimental setup

Datasets: technical reports of 91 LLMs released in 2025 (106 validated benchmarks)

Benchmarks: Humanity's Last Exam SimpleQA ZeroBench SWE-Bench-Verified AIME 2025 GSM8K MMLU-Pro HumanEval MBPP C-Eval

Metrics: BHI score Spearman rho Kendall tau max rank shift Pearson correlation

Baselines: BHI-EW (equal weighting)

Builds on: item response theory CRITIC weighting method coefficient of variation weighting

Closest papers by meaning

• BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.89)
• How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.89)
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.88)
• SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.88)
• tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.87)
• Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity (ICML 2025 · similarity 0.87)