Research map / Large language models / Alignment and preferences
Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory
Hongli Zhou, Hui Huang, Ziqing Zhao, Lvyuan Han, Huicheng Wang, Kehai Chen, Muyun Yang, Wei Bao, Jian Dong, Bing Xu, Conghui Zhu, Hailong Cao and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PSN-IRT instead of traditional IRT estimation (MLE, MCMC, VI, VIBO) | IRT 4PL · response prediction | accuracy: 0.7211 -> 0.7998 | Improved |
| use PSN-IRT 4PL instead of Deep-IRT | IRT estimation · rank stability | Kendall's tau: 0.9697 -> 1.0000 | Improved |
| replace one-hot inputs with Instructor semantic embeddings (ablation) | PSN-IRT · response prediction | accuracy: 0.7998 -> 0.7808 | Hurt |
| replace MLP backbone with a GNN (ablation) | PSN-IRT · rank stability | Kendall's tau: 1.0000 -> 0.7273 | Hurt |
| select 1000 items by Fisher information instead of using all items (ablation) | benchmark construction · without weak models | Kendall's tau vs arena ranking: 0.2381 -> 0.9048 | Improved |
| select 1000 items by discriminability instead of Fisher information (ablation) | benchmark construction · without weak models | score variance: 0.0030 -> 0.0000 | Hurt |
| select items by clustering success/failure vectors (ablation) | benchmark construction · without weak models | Kendall's tau vs arena ranking: 0.2381 -> 0.2381 | No clear effect |
| select 1000 items at random instead of all items (ablation) | benchmark construction · with weak models | Kendall's tau vs arena ranking: 0.6444 -> 0.6444 | No clear effect |
| increase Fisher-selected set size from 400 to 1000 items (ablation) | benchmark construction · without weak models | Kendall's tau vs arena ranking: 0.7143 -> 0.9048 | Improved |
| train PSN-IRT on 50% random item subset (ablation) | PSN-IRT · item parameter stability | Pearson correlation of difficulty with full-data estimate: 0.7437 (50%) vs 0.9519 (70%) | No clear effect |
Closest papers by meaning
- Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation (ICML 2026 · similarity 0.91)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.90)
- Reliable and Efficient Amortized Model-based Evaluation (ICML 2025 · similarity 0.90)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
