Research map / Large language models / AI and society

Can We Trust Item Response Theory for AI Evaluation?

NeurIPS 2026 · Large language models › AI and society · approach: Social-science view of AI evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Simulates AI-benchmark-like response matrices from six OpenLLM Leaderboard benchmarks to test whether 1PL/2PL/3PL IRT estimators (MML-EM, MCMC, VI, PSN) give reliable model rankings, item parameters, and short-form benchmarks under small model pools and skewed ability distributions.

Problem

IRT was developed for human testing with many examinees and few items, but AI benchmarks invert this (few models, many items) and have non-normal capability distributions, so the reliability of IRT-based AI evaluation is unverified.

Approach

A calibrated simulation study that generates response matrices from empirical item parameters and ability distributions of six LLM benchmarks, then systematically compares four estimators across IRT models and sample sizes on feasibility, ranking recovery, item recovery, and short-form benchmark compression.

Key results

• MML-EM failed in 69.45% of replications and ran out of memory on HellaSwag and MMLU (J>5,000); MCMC produced no finished runs for MMLU at N=1000 within 72 hours.
• Ranking recovery exceeded 0.85 for low skewness (|skew|<0.5) but fell below 0.60 for highly skewed conditions (|skew|>2.0) across all estimators.
• N=30 was insufficient for item parameter recovery under all estimators (difficulty recovery below 0.50 in some cases), with marked improvement at N=100.

Surprising findings

• VI, despite being widely used, showed unreliable difficulty recovery (below 0.5 for N<=180 on TruthfulQA and GSM8K under 2PL).
• Short-form benchmark ranking recovery did not depend monotonically on capability skewness and was insensitive to sample size and estimator.

Limitations

• Computational feasibility not measured on uniform hardware, so infeasibility thresholds are imprecise.
• Sample sizes between 30 and 100 not examined; simulations restricted to unidimensional IRT models and binary responses.
• No use of item content or semantic features; short-form benchmark drivers remain unclear.

Experimental setup

Datasets: ARC-Challenge HellaSwag MMLU TruthfulQA WinoGrande GSM8K OpenLLM Leaderboard v1 response matrices

Benchmarks: ARC-Challenge HellaSwag MMLU TruthfulQA WinoGrande GSM8K

Metrics: failure rate runtime Kendall tau ranking recovery aggregate score error difficulty recovery (Pearson) discrimination recovery guessing absolute error mean item error short-form ranking recovery (Kendall)

Baselines: MML-EM MCMC VI PSN random item subset

Builds on: item response theory ADEMP simulation framework py-irt mirt brms

Closest papers by meaning

• AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.91)
• Position: AI Evaluation Should Learn from How We Test Humans (ICML 2025 · similarity 0.88)
• Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation (ICML 2026 · similarity 0.87)
• Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation (NeurIPS 2026 · similarity 0.86)
• Position: Human Baselines in Model Evaluations Need Rigor and Transparency (With Recommendations & Reporting Checklist) (ICML 2025 · similarity 0.86)
• AGI-Elo: How Far Are We From Mastering A Task? (NeurIPS 2025 · similarity 0.85)