Research map / Large language models / Agents, code and math
metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills six Open LLM Leaderboard benchmarks (ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, WinoGrande) into a sparse 858-item benchmark using IRT latent abilities and information-based item filtering, reconstructing scores with about 1.24% RMSE.
Problem
Large LLM benchmarks are highly redundant and expensive to run; there is a need for a much smaller benchmark that still reconstructs the original individual and total benchmark scores accurately.
Approach
Uses item response theory (2PL/3PL/4PL) fitted on responses of over 5000 LLMs, selects items by Fisher information across ability quantiles, and jointly uses per-benchmark latent abilities in GAMs to reconstruct full scores, exploiting cross-benchmark synergy.
Key results
- Joint model reconstructs individual benchmark scores with mean test RMSE of 1.24% using 858 items (under 3% of original items).
- Reconstruction of the grand average Open LLM Leaderboard score reaches RMSE 0.582% and Spearman r = 0.998.
- A single factor captures 79.3% of variability across latent abilities, correlating r = 0.938 with the grand average score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| information filtering item selection (Fisher information, quantile-based) instead of random subsampling | IRT-based item selection per benchmark · six Open LLM Leaderboard benchmarks | test RMSE | Helped |
| joint model using latent abilities from all six benchmarks instead of single-benchmark marginal model | metabench score reconstruction · six benchmarks, Open LLM Leaderboard | test RMSE: mean RMSE 1.6% -> 1.24% | Helped |
| clustering-based item selection (k-means) instead of information filtering (ablation) | item selection · six benchmarks and Open LLM LB | test RMSE: Open LLM LB 0.582 -> 0.775; all benchmarks worse | Hurt |
| 3PL/4PL IRT vs 2PL with MAP latent abilities on 350 items (ablation) | IRT model choice · six benchmarks cross-validated score prediction | test RMSE | No clear effect |
| two-dimensional instead of one-dimensional IRT latent abilities (ablation) | score recovery GAM · six benchmarks, 350 items | test RMSE: deltas between -0.010 and 0.071 (mostly negligible) | No clear effect |
| logistic regression on normalized score instead of latent ability estimates (ablation) | score reconstruction · 350-item subsets | training RMSE: 4.06% vs approx 0.8-1.0% for latent abilities | Hurt |
| fine-tuning on metabench version A with fixed choices (ablation) | Llama 3.2-1B · metabench-A fixed choices | score gain (points): +7.6 on trained version, +1.3 permuted, +2.0 on version B | Helped |
| fine-tuning on metabench version A with fixed choices (ablation) | Llama 3.1-8B · metabench-B disjoint version | score gain (points): +0.4 fixed, -0.2 permuted | No clear effect |
Surprising findings
- Logistic regression score reconstruction using raw scores performs strictly worse than latent-ability estimates, even on the training set.
- Items with negative part-whole correlation were kept because correct responders scored worse on other items, contrary to common practice.
Limitations
- Static benchmark items may be memorized, with higher risk for smaller benchmarks.
- Conditional independence assumption of IRT likely violated because many LLMs are fine-tunes of each other from the same users.
- Models used for IRT were drawn from a leaderboard population, so generalization to new LLMs is assumed rather than proven.
Experimental setup
Datasets: ARC GSM8K HellaSwag MMLU TruthfulQA WinoGrande
Benchmarks: ARC GSM8K HellaSwag MMLU TruthfulQA WinoGrande Open LLM Leaderboard grand mean
Metrics: RMSE Spearman correlation MAE percent better than random
Baselines: random subsampling clustering-based item selection tinyBenchmarks logistic regression score reconstruction
Builds on: tinyBenchmarks BenchBuilder Item Response Theory Open LLM Leaderboard
Closest papers by meaning
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.92)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.92)
- MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.92)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.91)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.91)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.