Research map / Large language models / AI and society
Position: AI Evaluation Should Learn from How We Test Humans
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A position paper arguing that AI evaluation should adopt psychometric adaptive testing (IRT-based item characterization and per-model item selection) instead of static full-benchmark averaging.
Problem
Static benchmark evaluation of AI systems is costly, vulnerable to data contamination, and affected by low-quality or erroneous items, so averaged accuracy on fixed test sets may not reliably reflect model ability.
Approach
Import human psychometric tools (item response theory, Fisher-information item selection, cognitive diagnosis) into AI evaluation: annotate each benchmark item's difficulty, discrimination and guessing parameters from response data, then adaptively select a tailored subset of items per model to estimate its latent ability.
Key results
- In a simulation on MATH with 20 LLMs, Fisher-information adaptive selection reaches the highest ranking consistency with the full dataset using about 60% of items, and on the order of 20% of items for equivalent estimation accuracy (as claimed by the authors).
- Cited prior work reports selecting 50 of nearly 1,000 items gives 90% Kendall rank correlation with full test data.
- Cited prior work reports as little as 3% of items can reconstruct original benchmark scores across six benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive item selection with Fisher information instead of a randomly sampled test set | AI model ability estimation on MATH · MATH | items needed for the same estimation accuracy: only a maximum of 20% of the items | Helped |
| Use Fisher item selection with IRT ability estimates instead of evaluating with the full dataset | 20 LLMs · MATH | ranking consistency with full-dataset rankings: about 60% of the items | Helped |
| Use IRT ability estimates rather than traditional accuracy for random item selection | AI model ranking · MATH | rank correlation with full-dataset rankings | Helped |
| Select 50 items using Fisher Information | AI model ranking · Benchmark of nearly 1,000 items | Kendall's rank correlation with full test data: 90% | Helped |
| Select the most informative items using the Fisher method | AI benchmark evaluation · ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, WinoGrande | items needed to reconstruct original benchmark scores: 3% (or even fewer) | Helped |
| Reveal contaminated items and their answers in the model prompts | IRT item-characteristic estimates · MATH, NarrativeQA, RAFT | guessing factor | Hurt |
| Increase the temperature parameter | ChatGPT multiple-choice judgments · MATH | response entropy | Mixed |
Surprising findings
- ChatGPT answered the same multiple-choice item five times with the same prompt yielding four different answers, indicating response variability that psychometrics must model.
- Guessing factors estimated by IRT are significantly higher for items whose answers were revealed in the prompt (simulated contamination) than for clean items.
Limitations
- Mostly a position paper; the empirical illustrations are small toy or simulation experiments with limited detail.
- Validity of psychometric methods for AI systems is not yet established; the paper acknowledges need for further research or a new 'Machine Psychometrics' discipline.
- Item parameters depend on the response population of test-taker models, so annotations may not transfer across model groups.
Experimental setup
Datasets: MATH MedQA GSM8K SQuAD MMLU NarrativeQA RAFT
Benchmarks: MATH
Metrics: mean squared error of ability estimate Kendall rank correlation Jaccard similarity entropy of responses
Baselines: accuracy-based static benchmark evaluation
Builds on: item response theory computerized adaptive testing cognitive diagnosis models
Closest papers by meaning
- Can We Trust Item Response Theory for AI Evaluation? (NeurIPS 2026 · similarity 0.88)
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.88)
- Position: Human Baselines in Model Evaluations Need Rigor and Transparency (With Recommendations & Reporting Checklist) (ICML 2025 · similarity 0.87)
- EVALUATION CARDS: An Interpretive Layer for AI Evaluation Reporting (NeurIPS 2026 · similarity 0.86)
- AGI-Elo: How Far Are We From Mastering A Task? (NeurIPS 2025 · similarity 0.86)
- Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.