Research map / Large language models / Language, safety and interpretability
BenTo: Benchmark Reduction with In-Context Transferability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BenTo, which estimates task transferability with training-free in-context learning and selects a representative subset of benchmark tasks via facility location, reducing MMLU to about 5% of tasks with under 4% evaluation error.
Problem
Evaluating LLMs on large multi-task benchmarks is expensive; the paper asks how many tasks can be removed from a benchmark without degrading evaluation quality.
Approach
Measure pairwise task transferability by in-context learning (exemplars from a source task used as context for target-task queries), build a similarity graph with Laplacian Eigenmaps, and pick tasks by maximizing a facility location objective, which is submodular and solved greedily.
Key results
- On MMLU, BenTo-le reaches NRMSE 0.029 at k=3 (about 5% of 57 tasks), i.e. roughly 97% evaluation accuracy averaged over 9 models.
- On FLAN, BenTo-sim reaches NRMSE 0.04 at k=3 (about 5% of tasks), outperforming random (0.92 at k=3) and BM25 baselines.
- Reducing MMLU to 5% of tasks keeps per-model scores close to full-benchmark scores (e.g., Phi-3-mini 69.5 vs 70.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BenTo-sim task selection instead of random selection | LLM benchmark reduction · MMLU, best across k | NRMSE: 0.091 -> 0.026 | Helped |
| Use BenTo-le task selection instead of random selection | LLM benchmark reduction · FLAN, best across k | NRMSE: 0.49 -> 0.07 | Helped |
| Use BenTo rather than GPT4 task selection | LLM benchmark reduction · AGIEval English, best across k | NRMSE: 0.07 -> 0.03 | Helped |
| Use BenTo-le rather than GPT4 task selection | LLM benchmark reduction · Big Bench Hard, best across k | NRMSE: 0.072 -> 0.045 | Helped |
| Use BenTo rather than BM25 similarity on task selection | LLM benchmark reduction · Big Bench Hard, best across k | NRMSE: 0.032 -> 0.045 | Hurt |
| Replace Euclidean similarity with cosine similarity (ablation) | BenTo-le · MMLU, best across k | NRMSE: 0.03 vs 0.03 | No clear effect |
| Replace Euclidean similarity with Chebyshev similarity (ablation) | BenTo-sim · Big Bench Hard, best across k | NRMSE: 0.15 -> 0.22 | Hurt |
| Replace Euclidean similarity with cosine similarity (ablation) | BenTo-sim · Big Bench Hard, best across k | NRMSE: 0.15 -> 0.10 | Helped |
| Combine random example selection with BenTo task selection (ablation) | MMLU benchmark reduction · MMLU, 2.0% examples | NRMSE: 0.051 -> 0.026 | Helped |
| Combine random example selection with BenTo task selection (ablation) | MMLU benchmark reduction · MMLU, 0.7% examples | NRMSE: 0.109 -> 0.051 | Helped |
| Vary c from the paper's chosen value (ablation) | BenTo-le · MMLU | average NRMSE over k: 191; 340 | Mixed |
| Vary c from the paper's chosen value (ablation) | BenTo-le · FLAN | average NRMSE over k: 506; 872 | Mixed |
Surprising findings
- Using only task names, GPT-4 selection of tasks is competitive on FLAN but its selections are unstable across k (e.g., spike at k=7 on MMLU).
- Laplacian-eigenmap similarity (BenTo-le) is best for small k while Euclidean similarity (BenTo-sim) is better for larger k.
Limitations
- Smaller reduced benchmarks may be less diverse and more vulnerable to adversarial attacks.
- Choice of the constant c for the similarity kernel is empirically tuned and less optimal on FLAN.
- Few-shot BBH results show BM25 slightly better than the proposed method.
Experimental setup
Datasets: MMLU FLAN AGIEval English Big-Bench Hard
Benchmarks: MMLU FLAN AGIEval English Big-Bench Hard
Metrics: NRMSE accuracy perplexity-based metric on FLAN
Baselines: random GPT-4 task selection BM25-sim BM25-le K-medoids
Builds on: LEEP Laplacian Eigenmaps facility location BM25
Closest papers by meaning
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.89)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.88)
- MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models (ICLR 2025 · similarity 0.88)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.87)
- How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.