Research map / Large language models / Reasoning and chain of thought

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ATLAS, an IRT-based computerized adaptive testing framework that estimates LLM ability with Fisher-information item selection and SE-based stopping, cutting evaluation items by up to 90% while preserving accuracy reconstruction across five benchmarks.

Problem

LLM evaluation relies on large fixed benchmarks and average accuracy, which treats all items as equally informative, is inefficient, and obscures ability differences among models with identical accuracy.

Approach

Calibrates 3PL IRT item parameters on the Open LLM Leaderboard response matrix using partition-based common-person linking, then adaptively selects items by Fisher information (randomesque top-5) with EAP updates and SE-controlled stopping, reporting ability estimates alongside p-IRT reconstructed accuracy.

Key results

• ATLAS matches whole-bank ability on HellaSwag with 41 items (MAE 0.157) and achieves the lowest MAE on TruthfulQA (0.064 with 48 items), reducing items by up to 90%.
• Ability-based rankings have higher split-half Spearman correlation than accuracy (e.g., WinoGrande 0.981 vs 0.943; GSM8K 0.993 vs 0.985).
• 23-31% of models shift by more than 10 rank positions when ranked by IRT ability instead of accuracy; ATLAS accuracy reconstruction MAE stays below 5%.

Surprising findings

• TinyBenchmarks shows severe model misfit (RMSEA in the hundreds), and MetaBench shows poor fit on TruthfulQA.
• Temporal generalization degrades ability-estimation MAE from 0.084-0.117 to 0.126-0.162 on ARC.

Limitations

• Calibration relies on Open LLM Leaderboard responses from a specific model population; generalization to newer or unseen families is only partially validated (temporal MAE increases).
• Ability estimates use a precision-based stopping rule and a 3PL model; 2PL comparison shows benefits are benchmark-dependent.
• Evaluation uses only five benchmarks and reconstructed accuracy metrics rely on calibrated IRT assumptions.

Experimental setup

Datasets: WinoGrande TruthfulQA HellaSwag GSM8K ARC HuggingFace Open LLM Leaderboard responses MMLU Elementary Math MMLU College Chemistry MMLU High School Chemistry

Benchmarks: WinoGrande TruthfulQA HellaSwag GSM8K ARC

Metrics: MAE SE of absolute error average test length IES Spearman correlation RMSEA item exposure rate test overlap rate

Baselines: Random_100 TinyBenchmarks MetaBench-Primary MetaBench-Secondary

Builds on: computerized adaptive testing 3PL IRT TinyBenchmarks MetaBench Fluid Benchmarking

Closest papers by meaning

• Reliable and Efficient Amortized Model-based Evaluation (ICML 2025 · similarity 0.91)
• tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.90)
• Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing (ICML 2025 · similarity 0.90)
• RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.90)
• An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.89)
• LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.89)