Research map / Large language models / Reasoning and chain of thought
On Speeding Up Language Model Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UCB-E and UCB-E-LRF, bandit-based adaptive selection of (method, validation example) pairs with low-rank matrix completition, to identify the best LLM prompt/config using a small fraction of the full evaluation budget.
Problem
Identifying the best LLM-based method (prompt, model, or hyperparameter setting) across a validation set is combinatorially expensive and costly in LLM inference and LLM-judge calls under a fixed evaluation budget.
Approach
Treat method selection as fixed-budget best-arm identification: UCB-E picks the method with the highest upper confidence bound and a random unseen example; UCB-E-LRF additionally fills missing scores with an ensemble low-rank factorization and selects the example with highest predictive uncertainty.
Key results
- UCB-E and UCB-E-LRF reach top-1 precision 1 on AlpacaEval (Drop Annotator) with about 8% of the budget, while baselines need 80-90%.
- The proposed methods identify the top method using only 5-15% of typical resources, per the abstract, yielding 85-95% LLM cost savings.
- On harder prompt-tuning datasets (GSM8K Prompts, PIQA Prompts), UCB-E-LRF reaches higher precision faster than UCB-E, saving about 10% absolute budget.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use active selection with UCB-E or UCB-E-LRF instead of non-active baselines | Best-method identification · AlpacaEval (Drop Annotator) | evaluation budget to reach precision 1: 8% vs 80-90% | Helped |
| Use UCB-E rather than UCB-E-LRF | Best-method identification · AlpacaEval | evaluation budget: 2-3% | Helped |
| Use UCB-E-LRF rather than UCB-E | Best-method identification · GSM8K Prompts and PIQA Prompts | evaluation budget to achieve higher precision: about 10% in absolute budget | Helped |
| Remove uncertainty matrix R from UCB-E-LRF (Score Only) (ablation) | UCB-E-LRF · GSM8K Prompts | top 1 precision | Hurt |
| Reduce batch size from 32 to 1 (ablation) | UCB-E-LRF · Harder datasets | performance | Helped |
| Increase batch size from 32 to 128 (ablation) | UCB-E-LRF · Datasets | performance | Hurt |
| Increase batch size from 32 to 1 (ablation) | UCB-E-LRF · Datasets | time to achieve precision 1: almost 16 times | Hurt |
| Reduce ensemble size to 4 (ablation) | UCB-E-LRF · Hard datasets | performance | Hurt |
| Vary ensemble size (ablation) | UCB-E-LRF · AlpacaEval (Drop Annotator) | performance | No clear effect |
| Use ensemble size larger than 64 (ablation) | UCB-E-LRF · Datasets | performance | No clear effect |
| Vary uncertainty scaling from 0.1 to 5 (ablation) | UCB-E-LRF · Two evaluated datasets | performance: 0.1 to 5 | No clear effect |
| Reduce warm-up budget to 0.5% (ablation) | UCB-E-LRF · Two evaluated datasets | precision as more data are evaluated: 0.5% | Hurt |
Surprising findings
- Row mean imputation and filled subset baselines need 90% of method-example pairs on some datasets to predict the best method reliably.
- Rank-1 factorization was consistently better than higher ranks under limited budgets.
Limitations
- Assumes a fixed-size two-dimensional scoring matrix; dynamic rows/columns not handled.
- Does not directly address pairwise settings such as Chatbot Arena.
- Targets only top-1 identification rather than full or top-K ranking.
Experimental setup
Datasets: AlpacaEval GSM8K PIQA
Benchmarks: AlpacaEval GSM8K PIQA
Metrics: top-1 precision (performance gap) top-1 precision (McNemar significance) NDCG@10
Baselines: Full Evaluation Row Mean Imputation Filled Subset LRF
Builds on: UCB-E successive elimination alternating least squares AlpacaEval
Closest papers by meaning
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.91)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.90)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)
- Online Prompt Selection for Program Synthesis (AAAI 2025 · similarity 0.89)
- Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.88)
- Efficient Evaluation of LLM Performance with Statistical Guarantees (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.