Research map / Large language models / Language, safety and interpretability
Efficient multi-prompt evaluation of LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptEval, an IRT-based (X-pIRT) method that estimates the performance distribution and quantiles of an LLM across a large pool of prompt templates from a small budget of evaluations, with consistency guarantees.
Problem
Evaluating LLMs over many prompt templates to get a performance distribution is prohibitively expensive, since exact per-template scores need I*J evaluations, and single-prompt leaderboards are unreliable.
Approach
Model correctness as a logistic function of prompt and example parameters (Rasch/linear IRT with optional prompt covariates like embeddings), fit by logistic regression on a two-way balanced sample of prompt-example pairs, and plug the fitted model into a performance-IRT conditional expectation to estimate every template's score, then derive quantiles from these estimates.
Key results
- On MMLU, PromptEval estimates performance quantiles across 100 prompt templates with a budget equivalent to about two single-prompt evaluations, with median estimates accurate with 200 evaluations (over 100x compute saving in most cases).
- PE variants (including the simple Rasch version) achieve much lower Wasserstein-1 distance than the avg baseline across BBH, LMentry, and MMLU at budgets of 200-1600 evaluations.
- In LLM-as-a-judge with GPT-4o-mini, the ranking of four LLMs changes in 36% of the 100 judge prompt templates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PromptEval instead of the avg baseline for distribution estimation | PromptEval · MMLU, BBH, and LMentry | Wasserstein-1 distance | Helped |
| Use PromptEval instead of the avg baseline for quantile estimation | PromptEval · MMLU, BBH, and LMentry | quantile estimation error | Helped |
| Add prompt covariates to PE-Rasch (ablation) | PromptEval · MMLU, BBH, and LMentry | Wasserstein-1 distance | Helped |
| Use pre-trained prompt embeddings (PE-EmbPT) (ablation) | PromptEval · MMLU, BBH, and LMentry | quantile estimation error | Helped |
| Use discrete prompt covariates (PE-discrete) (ablation) | PromptEval · LMentry | quantile estimation error | Hurt |
| Fine-tune BERT to produce prompt embeddings (PE-EmbFT) (ablation) | PromptEval · MMLU, extreme quantiles with small evaluation budgets | quantile estimation error | Helped |
| Use central quantiles rather than extreme quantiles | PromptEval · MMLU, BBH, and LMentry | quantile estimation error | Mixed |
| Use PromptEval rather than avg to estimate judge score distributions | PromptEval · AlpacaEval 2.0 | Wasserstein-1 loss | Helped |
| Use PromptEval for best-prompt identification rather than TRIPLE-GSE | PromptEval · MMLU, BBH, and LMentry | regret | Helped |
| Reduce the prompt pool to 20% of its original size (ablation) | PromptEval · MMLU | performance estimation quality | Hurt |
| Represent examples with sentence-transformer covariates instead of one-hot encodings (ablation) | PromptEval · Preliminary tests | estimation performance | No clear effect |
| Use two-way balanced sampling rather than uniform sampling without replacement | PromptEval · Multi-prompt evaluation | sampling stability | Helped |
Surprising findings
- Across LLMs there is no coherent template preference; within MMLU subjects Kendall's W is low for most models, with gemma-7b-it the highest at 0.45.
- Example covariates (sentence-transformer embeddings) did not improve over one-hot example encodings in preliminary tests.
Limitations
- Requires a predefined pool of prompt templates; does not address prompt generation or engineering.
- Extreme quantiles are hard to estimate and need more evaluations.
- Fine-tuned embeddings require collecting large amounts of evaluation data from other LLMs.
Experimental setup
Datasets: MMLU BIG-bench Hard LMentry AlpacaEval 2.0
Benchmarks: MMLU BIG-bench Hard LMentry AlpacaEval 2.0
Metrics: Wasserstein-1 distance quantile estimation error regret
Baselines: avg (simple averaging over evaluated examples) TRIPLE-GSE TRIPLE-MLP-GSE
Builds on: Item Response Theory Performance-IRT (pIRT) Rasch model linear logistic test model sentence transformers
Closest papers by meaning
- Towards Reliable LLM Evaluation: Correcting the Winner’s Curse in Adaptive Benchmarking (NeurIPS 2026 · similarity 0.89)
- Prompt-to-Leaderboard: Prompt-Adaptive LLM Evaluations (ICML 2025 · similarity 0.88)
- Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions (AAAI 2025 · similarity 0.88)
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark (ICLR 2025 · similarity 0.88)
- From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set (ICML 2025 · similarity 0.87)
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.