Research map / Large language models / Reasoning and chain of thought
Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R-AutoEval+, an adaptive testing-by-betting framework that mixes real and LLM-judge data via a learned reliance factor, giving finite-sample reliability and sample-efficiency no worse than R-Eval and R-AutoEval.
Problem
Reliable model selection/risk certification using LLM-as-judge autoevaluators: autoevaluation with PPI can reduce variance but may hurt sample efficiency and has only asymptotic guarantees when the judge is poor.
Approach
Introduces a PPI++-style effective observation with a reliance factor rho, runs several rho candidates in parallel, and combines their e-values with exponential weights updated by accumulated evidence, so the test adaptively reverts to real-data-only testing when the autoevaluator is inaccurate.
Key results
- On LLM quantization (TriviaQA, alpha=0.1), R-AutoEval+ selects smaller models than R-Eval and R-AutoEval when the autoevaluator is BF16/MX6 and recovers R-Eval with the low-quality MX4 judge.
- On reasoning budget selection for Qwen3-1.7B on GSM8K, R-AutoEval+ saves up to 127 tokens over R-Eval and up to 66 tokens over R-AutoEval on average (e.g., 856.13 vs 883.99 tokens with GPT-4.1 judge).
- Across all tested setups, R-AutoEval+ yields the smallest confidence intervals and consistently outperforms R-Eval and R-AutoEval in the prompt-selection task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adaptively weight synthetic data rather than using R-Eval alone | R-AutoEval+ with GPT-4.1 · GSM8K | average generated tokens: 983.34 (137.87) -> 856.13 (70.93) | Helped |
| adaptively weight synthetic data rather than using R-AutoEval | R-AutoEval+ with Qwen3-32B · GSM8K | average generated tokens: 1007.45 (150.48) -> 941.20 (129.61) | Helped |
| use S=5 instead of S=2 candidate reliance factors (ablation) | R-AutoEval+ with GPT-4.1 · GSM8K | average generated tokens: 871.67 (16.24) -> 854.99 (14.02) | Helped |
| increase candidate reliance factors from S=10 to S=20 (ablation) | R-AutoEval+ with GPT-4.1 · GSM8K | average generated tokens: 856.13 (13.90) -> 856.13 (13.90) | No clear effect |
| replace equally spaced reliance factors with Dir(1,...,1) factors (ablation) | R-AutoEval+ with GPT-4.1 · GSM8K | average generated tokens: 856.13 (13.90) -> 860.91 (15.63) | Hurt |
| replace equally spaced reliance factors with Dir(0.1,...,0.1) factors (ablation) | R-AutoEval+ with GPT-4.1 · GSM8K | average generated tokens: 856.13 (13.90) -> 873.27 (15.10) | Hurt |
| replace equally spaced reliance factors with Dir(1,...,1) factors (ablation) | R-AutoEval+ with DeepSeek-R1-0528-Qwen3-8B · GSM8K | average generated tokens: 961.94 (27.24) -> 960.76 (27.09) | Helped |
| replace equal initial weights with Dir(10,...,10) weights (ablation) | R-AutoEval+ with DeepSeek-R1-0528-Qwen3-8B · GSM8K | average generated tokens: 961.94 (27.24) -> 959.21 (27.04) | Helped |
| replace equal initial weights with Dir(0.1,...,0.1) weights (ablation) | R-AutoEval+ with DeepSeek-R1-0528-Qwen3-8B · GSM8K | average generated tokens: 961.94 (27.24) -> 969.50 (25.59) | Hurt |
| use adaptive reliance factors rather than R-AutoEval alone | R-AutoEval+ · GSM8K | normalized deviation of generated tokens: 3.90% -> 6.88%; 6.32% -> 6.00% | Mixed |
Surprising findings
- Same-family autoevaluators (e.g., Qwen3-32B for Qwen3-1.7B) can make R-AutoEval require more tokens than R-Eval, attributed to preference leakage.
- A higher-accuracy autoevaluator is not always better: Llama-3.2-3B-Instruct (66%) helped reduce tokens more than Qwen3-32B (82%) for R-AutoEval due to family effects.
Limitations
- Requires access to real-world unlabeled data.
- The set of candidate reliance factors is fixed a priori.
- Sample-efficiency guarantee holds only for sufficiently high target reliability levels (1-delta).
Experimental setup
Datasets: TriviaQA Instruction-Induction GSM8K CoQA
Benchmarks: TriviaQA Instruction-Induction GSM8K CoQA
Metrics: model size (selected) performance drop execution accuracy tokens generated (sample efficiency) confidence interval width
Baselines: R-Eval R-AutoEval Eval AutoEval fixed sequence testing Bonferroni correction
Builds on: prediction-powered inference PPI++ testing-by-betting R-AutoEval R-Eval
Closest papers by meaning
- Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks (ICLR 2025 · similarity 0.90)
- AutoEval Done Right: Using Synthetic Data for Model Evaluation (ICML 2025 · similarity 0.88)
- From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set (ICML 2025 · similarity 0.88)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.87)
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.87)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.