Research map / Large language models / Language, safety and interpretability
How Benchmark Prediction from Fewer Data Misses the Mark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically benchmarks 11 benchmark prediction methods on 19 benchmarks, finding a random-sample-plus-regression baseline competitive under interpolation but all methods (except random sampling and a new AIPW-based estimator) fail under extrapolation to stronger models.
Problem
Benchmark prediction (efficient LLM evaluation) selects a small core set of evaluation points and predicts full-benchmark performance; it is unclear how well these methods work and whether they generalize to new, stronger models.
Approach
Shows that simply fitting a ridge regression on a random sample predicts full-benchmark performance nearly as well as sophisticated core-set selection under interpolation, and introduces an AIPW-style estimator for benchmark prediction that stays consistent under extrapolation, while documenting that all methods' accuracy depends on model similarity.
Key results
- Random-Search-Learn reduces average estimation gap by 42.1% vs Random-Sampling under interpolation (n=50), versus 29.9% for GP-IRT; Random-Sampling-Learn reduces it by 37.2%.
- Under extrapolation, Random-Search-Learn increases estimation gap by 185.1% over Random-Sampling, while AIPW is the only method beating Random-Sampling on average (-13.8% gap reduction vs -32.3% in interpolation).
- Random-Sampling with twice the data (n=100) outperforms all other benchmark prediction methods under interpolation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fit a Ridge regression model on a random core set instead of directly averaging its scores | Random-Sampling · Interpolation, across 19 benchmarks | average estimation gap: 37.2% reduction | Helped |
| Use cross-validation to select among 10,000 random core sets | Random-Sampling-Learn · Interpolation, across 19 benchmarks | average estimation gap: 42.1% reduction | Helped |
| Use AIPW rather than random-sample mean estimation | Random-Sampling · Interpolation, across 19 benchmarks | average estimation gap: -32.3% | Helped |
| Use AIPW rather than random-sample mean estimation | Random-Sampling · Extrapolation, across 19 benchmarks | average estimation gap: -13.8% | Helped |
| Use Random-Search-Learn rather than random-sample mean estimation | Random-Sampling · Extrapolation, across 19 benchmarks | average estimation gap: 185.1% increase | Hurt |
| Increase the core-set size from n=50 to n=100 (ablation) | Random-Sampling · Interpolation, average across benchmarks | estimation gap: 4.8 -> 3.3 | Helped |
| Increase the core-set size from n=50 to n=100 (ablation) | Random-Sampling · Extrapolation, average across benchmarks | estimation gap: 4.6 -> 3.1 | Helped |
| Use Random-Sampling with n=100 instead of AIPW with n=50 | ImageNet models · ImageNet, target models with low source-model similarity | normalized estimation gap | Helped |
| Use twice as much data for Random-Sampling | Random-Sampling · Interpolation | estimation gap | Helped |
Surprising findings
- A simple random sample with ridge regression beats most sophisticated core-set selection methods under interpolation, suggesting core-set selection matters little.
- Random-Sampling with 2x the evaluation budget outperforms all other benchmark prediction methods even in the interpolation setting.
Limitations
- AIPW gains are modest and still depend on model similarity; improvements diminish when target accuracy exceeds source accuracy.
- Extrapolation setting relies on ranking models by full-benchmark accuracy, which is a specific simulated split rather than true future models.
- P-IRT and GP-IRT only applicable to binary (accuracy) metrics, restricting benchmark selection.
Experimental setup
Datasets: OpenbookQA GSM8K LegalBench MATH MedQA MMLU MRPC RTE SST-2 MNLI QNLI IFEval
Benchmarks: HELM-Lite GLUE OpenLLM Leaderboard ImageNet
Metrics: average estimation gap normalized estimation gap estimation gap reduction over Random-Sampling Kendall's tau Pearson correlation with model similarity
Baselines: Random-Sampling Anchor-Points-Weighted Anchor-Points-Predictor P-IRT GP-IRT PCA Lasso Double-Optimize
Builds on: Anchor Points P-IRT GP-IRT prediction-powered inference augmented inverse propensity weighting
Closest papers by meaning
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.91)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
- Training on the Benchmark Is Not All You Need (AAAI 2025 · similarity 0.90)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.90)
- Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (ICLR 2026 · similarity 0.89)
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.