Research map / Large language models / AI and society
Expanding the AI Evaluation Toolbox with Statistical Models
Drew Keller, Kweku Kwegyir-Aggrey, Ryan Steed, Anita K. Rao, Julia L. Sharp, A. Stevie Bergman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a GLMM instead of regression-free estimator for generalized accuracy | generalized accuracy estimation · simulated toy settings and GPQA Diamond | confidence interval width | Improved |
| use a GLMM instead of regression-free estimator when GLMM is well specified | generalized accuracy estimation · simulated Setting A | RMSE and coverage | No clear effect |
| misspecify the GLMM data-generating process (ablation) | GLMM generalized accuracy estimates · simulated misspecification | long-run coverage: coverage in the 80-90% range | Hurt |
| increase number of items n (ablation) | GLMM vs regression-free CI width · simulated toy settings | relative CI width advantage of GLMM | Hurt |
| add more trials per item (ablation) | regression-free estimator · GPQA Diamond generalized accuracy | confidence interval width | Improved |
| ignore item effects (fixed-effect-only model) (ablation) | GLMM variance estimates · BIG-Bench Hard | generalized accuracy variance | Hurt |
| use simple average with standard error for benchmark accuracy | benchmark accuracy CIs · GPQA Diamond | CI calibration | Hurt |
Closest papers by meaning
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.91)
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (ICLR 2025 · similarity 0.89)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.89)
- Fantastic Bugs and Where to Find Them in AI Benchmarks (NeurIPS 2025 · similarity 0.89)
- RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts (ICML 2025 · similarity 0.88)
- Can We Trust Item Response Theory for AI Evaluation? (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
