Research map / Large language models / Reasoning and chain of thought

Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a semiparametric one-step (EIF-based) estimator that uses target-model pairwise judgments of auxiliary reasoning chains as control variates to reduce variance of LLM accuracy estimates on small math/science benchmarks.

Problem

Accuracy estimates of LLMs on mathematical and expert reasoning benchmarks are high-variance and rankings are unstable when benchmarks are small (e.g., AIME 2025 with 30 problems), and repeated sampling of final answers alone does not improve efficiency.

Approach

Treat the target model's pairwise preference over auxiliary model reasoning chains as auxiliary variables (control variates) and combine them with the naive accuracy estimate through an efficient-influence-function one-step estimator with cross-fitting, which is shown to be asymptotically normal, semiparametrically efficient, and strictly variance-reducing; for tiny samples an off-the-shelf LLM is

Key results

• On AIME 2025 (N=15), the one-step estimator moves DeepSeek-V3.2 (Thinking) estimate from 86.67% (naive) to 91.00% under Config 1, versus 90.00% ground truth.
• On GPQA Diamond (N=50), the one-step estimator is closer to ground truth than the naive estimator for most models, e.g., Llama-3.3-70B improves from 32.00% naive to 37.60% (GT 46.97%) under Config 1.
• In simulations with K=5 cross-fitting and M=500, the one-step estimator's ranking accuracy stays stable as model output noise grows while the naive estimator's declines sharply.

Surprising findings

• On AIME 2025 with Claude-Sonnet-4.5 under Config 2, the one-step estimator gives 78.67% vs naive 66.67% against 83.33% ground truth, but for Qwen3-Next-80B under Config 2 the one-step estimate is worse than naive (-5.33 points).

Limitations

• Gains are not uniform: several cases show the one-step estimate further from ground truth than the naive estimate (e.g., Qwen3-Next-80B on AIME under Config 2, Gemini-3-Flash on GSM8K Config 2).
• Ground truth is itself a naive estimate on the full dataset, so 'improvement' is measured against a noisy reference.
• Theory assumes the auxiliary generation distribution p(z|x) is known; the small-sample variant uses an off-the-shelf LLM regressor without formal guarantees.

Experimental setup

Datasets: GPQA Diamond AIME 2025 GSM8K

Benchmarks: GPQA Diamond AIME 2025 GSM8K

Metrics: accuracy estimation error vs ground truth exact match ranking accuracy Kendall's tau variance reduction ratio

Baselines: naive sample-mean accuracy estimator repeated decoding of target answers (repeated-Y)

Builds on: semiparametric efficient influence function theory cross-fitting (double machine learning)

Closest papers by meaning

• Evaluating Mathematical Reasoning Beyond Accuracy (AAAI 2025 · similarity 0.91)
• Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.90)
• Statistical Early Stopping for Reasoning Models (ICML 2026 · similarity 0.90)
• MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.89)
• CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.89)
• Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist (ICLR 2025 · similarity 0.89)