Research map / Large language models / Agents, code and math
tinyBenchmarks: evaluating LLMs with fewer examples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLM benchmark scores can be estimated within about 2% error using ~100 curated examples per scenario, selected by clustering and IRT-based estimators, and releases tiny benchmarks.
Problem
Evaluating LLMs on large benchmarks (tens of thousands of examples, or GPT-4 judged) is expensive in compute, money, and environment, and evaluation is repeated often during development.
Approach
Select a small set of anchor examples (via clustering on correctness patterns or IRT embeddings of examples) and combine their observed correctness with an IRT model's predicted probabilities on the remaining examples (gp-IRT, the "++" estimator) to estimate full-benchmark performance.
Key results
- On MMLU, IRT++ with 100 examples (out of 14K) estimates accuracy within about 2% average error.
- On Open LLM Leaderboard, 30 examples per scenario (180 total of 29K) suffices, a ~160x cost reduction.
- The gp-IRT (++) variant always improves or matches its vanilla counterpart; on specialized MMLU models correctness-based anchors degrade while IRT anchors remain robust.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IRT-based anchors with gp-IRT rather than evaluating the full benchmark | LLM performance evaluation · MMLU | performance estimation error: within 1.9% | Helped |
| Use IRT++ with 100 or fewer examples per dataset or scenario | LLM performance evaluation · Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0 | performance estimation error: within 2% | Helped |
| Add the gp-IRT adjustment to the vanilla IRT estimator (ablation) | IRT evaluation strategy · All four benchmarks | performance estimation error | Helped |
| Replace correctness-based anchors with IRT-based anchors | MMLU evaluation of specialized LLMs · MMLU | performance estimation error | Helped |
| Use correctness-based anchors on specialized models instead of the random-model setting | Correctness-based anchor strategy · MMLU | performance estimation error | Hurt |
| Increase stratified random sample size to 400 examples per task | Open LLM Leaderboard evaluation · Open LLM Leaderboard | examples needed to match IRT++: 400 examples per task (2400 total) ... IRT++ with 100 examples per task (600 total) | Hurt |
| Increase stratified random sample size to 200 examples | AlpacaEval 2.0 evaluation · AlpacaEval 2.0 | examples needed to match IRT++: random with 200 examples matches IRT++ with 100 examples | Hurt |
| Use adaptive IRT testing instead of preselected examples (ablation) | LLM performance evaluation · MMLU | performance estimation error | Helped |
Surprising findings
- Correctness-based anchor selection deteriorates notably on specialized LLMs, while IRT-based anchors are only slightly affected.
- Stratified random sampling needs over 400 examples on MMLU to match IRT++ with 100 examples.
Limitations
- Performance degrades under severe distribution shifts that change correctness patterns (e.g., models failing simple items but solving complex ones).
- Rapid capability increases may cause extrapolation errors, requiring periodic updates of anchors and IRT parameters.
- Adaptive IRT variant takes over 5 minutes per run, less practical.
Experimental setup
Datasets: Open LLM Leaderboard (6 scenarios, ~29K examples) MMLU (~14K examples) HELM Lite v1.0.0 (10 scenarios, ~10K examples) AlpacaEval 2.0 (805 examples) ANLI prompt-template evaluations (LLaMA)
Benchmarks: Open LLM Leaderboard MMLU HELM Lite AlpacaEval 2.0
Metrics: absolute estimation error Spearman rank correlation
Baselines: stratified random sampling correctness-based anchor points p-IRT
Builds on: Anchor Points Item Response Theory stratified sampling for HELM
Closest papers by meaning
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.93)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.93)
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.92)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.92)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.92)
- Benchmarking LLMs via Uncertainty Quantification (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.