Research map / Large language models / Reasoning and chain of thought

Towards Reliable LLM Evaluation: Correcting the Winner’s Curse in Adaptive Benchmarking

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SIREN, a repeated-split reporting protocol with a Gaussian multiplier bootstrap that estimates the fresh-data performance of budget-indexed tune-then-deploy procedures, removing winner's-curse optimism in MMLU-Pro tuning experiments.

Problem

Reporting the best artifact found by adaptive prompt/program search on reused benchmark items is optimistic; the fresh-data performance of the full budgeted tune-then-deploy procedure is needed instead.

Approach

Freeze the post-search shortlist, repeatedly split the evaluation pool into scoring and held-out parts, use softmax-type selector weights from scoring halves evaluated on held-out halves, and bootstrap item-level first-order contributions (including selection derivative terms) with Gaussian multipliers, avoiding tuner reruns.

Key results

• SIREN's mean signed bias on MMLU-Pro Math/Law is between -0.20 and +0.08 pp, versus +1.37 to +2.16 pp for naive max and R-split t reporting.
• SIREN gets directional agreement with the Monte Carlo reference in 42/44 (Math random search), 43/44 (Math DSPy) and 40/44 (Law DSPy) cells, outperforming M1-M4 in every summary.
• PromptEval's error changes sign with observation fraction f (about -8 pp at f=0.05 to +1.5 pp at f≥0.2), while SIREN stays within 0.1 pp.

Surprising findings

• Sparse PromptEval can achieve high directional accuracy through shrinkage while its point estimates are substantially biased.
• Single-split holdout (M3) sometimes produces negative bias under DSPy, i.e. it can be pessimistic rather than optimistic.

Limitations

• Theory holds only in a fixed-shortlist, smooth-selector regime; hard argmax requires smoothing or a margin condition.
• Guarantees are conditional on the completed tuning stage and do not cover open-ended adaptive search.
• Experiments are limited to MMLU-Pro Math and Law with 11 open-weight models and two tuners.

Experimental setup

Datasets: MMLU-Pro

Benchmarks: MMLU-Pro

Metrics: signed bias (percentage points) directional accuracy confidence interval coverage

Baselines: naive max (M1) M1 + Wald interval (M2) single-split holdout (M3) R-split argmax with Student-t CI (M4) PromptEval

Builds on: DSPy PromptEval Gaussian multiplier bootstrap

Closest papers by meaning

• Efficient multi-prompt evaluation of LLMs (NeurIPS 2024 · similarity 0.89)
• Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (ICLR 2026 · similarity 0.88)
• Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.87)
• NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric (ICML 2025 · similarity 0.86)
• Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scaling (ICLR 2026 · similarity 0.86)
• Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators (ICML 2025 · similarity 0.85)