Research map / Large language models / Reasoning and chain of thought

Efficient Evaluation of LLM Performance with Statistical Guarantees

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Factorized Active Querying (FAQ), combining a Bayesian logistic factor model over historical LLM benchmark outcomes with a hybrid variance-reduction/active-learning sampling policy and a finite-population "Pro-Active Inference" estimator that keeps frequentist CI coverage while cutting queries needed for LLM benchmark evaluation.

Problem

Evaluating many LLMs on large benchmark banks is expensive; the goal is to estimate each new model's finite-bank accuracy with a fixed query budget while producing narrow confidence intervals that maintain valid frequentist coverage.

Approach

Use a partially observed historical model-by-question outcome matrix to fit logistic factor models, then adaptively select questions via a tempered mix of oracle variance-reduction (sqrt(p(1-p))) and active-learning scores, and form an inverse-propensity-weighted, model-assisted estimator (PAI) that is unbiased and asymptotically normal with model-free coverage, enabling proactive question selecti

Key results

• On fully observed history, FAQ achieves 4-5x higher ESS than uniform sampling and 1.8-2.4x higher ESS than the strongest post-hoc baseline, with coverage near 95%.
• Cold-start FAQ on GPQA (All) with zero target-suite history reaches ESS multipliers up to 1.32x over uniform sampling and beats the strongest baseline with 0.1% target history on 8 of 10 budgets.
• One parallelized FAQ run over 2.2K models and 12K questions takes under one minute on a single H100 GPU.

Surprising findings

• Sequential active-inference ablation can match or outperform FAQ at larger budgets.
• Even with only 0.1% of historical entries observed, FAQ retains a modest ESS edge at most budgets, though it degrades at the two largest budgets.

Limitations

• Mild ESS degradation and slight miscoverage (under 1%) at the smallest budgets due to noisy early factor estimates and CLT approximation.
• Evaluated only on binary per-question outcomes from two suites of leaderboard benchmarks; interactive agent benchmarks are not evaluated.
• Post-hoc baseline selection favors baselines; FAQ hyperparameters are tuned on validation models.

Experimental setup

Datasets: MMLU-Pro BBH GPQA IFEval MATH MuSR Open LLM Leaderboard

Benchmarks: MMLU-Pro BBH+GPQA+IFEval+MATH+MuSR GPQA (All) cold-start

Metrics: effective sample size empirical coverage of 95% CIs CI width

Baselines: uniform sampling AIPW with Bernoulli sampling Neyman-style AIPW sampling sequential active inference

Builds on: active statistical inference Bayesian active learning Laplace approximation for Bayesian logistic regression AIPW estimator

Closest papers by meaning

• Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.90)
• MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.90)
• Scaling Up Active Testing to Large Language Models (NeurIPS 2025 · similarity 0.90)
• Reliable and Efficient Amortized Model-based Evaluation (ICML 2025 · similarity 0.89)
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.89)
• tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)