Research map / Large language models / AI and society
AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems
Michael Hardy, Anka Reuel, Lijin Zhang, Jodi M. Casabianca, Sang T. Truong, Yash Satish Dave, Hansol Lee, Benjamin W. Domingue, Sanmi Koyejo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fit a bifactor structure (general plus benchmark-specific factors) instead of independent or unidimensional structures (ablation) | CFA of Open LLM Leaderboard items · held-out AUC | AUC: 0.90 -> 0.95 | Improved |
| allow correlated specific factors on top of the bifactor (correlated bifactor) (ablation) | bifactor CFA · held-out AUC | AUC: 0.95 -> 0.77 | Hurt |
| allow correlated specific factors on top of the bifactor (correlated bifactor) (ablation) | bifactor CFA · RMSEA | RMSEA: 0.05 -> 0.04 | Improved |
| randomize item-to-benchmark assignment (permutation control) (ablation) | flexible SEM structures · item-level CFA | fit | No clear effect |
| add size random slopes and control for model size in the variance decomposition (ablation) | G-study · Open LLM Leaderboard | contributor variance share: 0.086 -> 0.078 | No clear effect |
| estimate scaling slope on manifest scores instead of latent general factor | scaling law · Open LLM Leaderboard | slope reliability: 0.97 -> 0.53 | Hurt |
| control for deployment and contributor effects in latent regression of g on size (ablation) | bifactor latent scores · general factor scaling | size slope | Improved |
| increase model size (log10 params), after controls | MMLU-Pro specific factor · latent regression | slope beta: beta 0.52 | Improved |
| increase model size (log10 params), after controls | BBH and GPQA specific factors · latent regression | slope beta: BBH -0.05, GPQA 0.07 | No clear effect |
| increase model size (log10 params), after controls | MuSR specific factor · latent regression | slope beta: beta 0.04 | No clear effect |
| apply chat template / instruction fine-tuning | IFEval specific factor · latent regression | effect gamma: gamma 1.82 | Improved |
| apply chat template / instruction fine-tuning | MuSR specific factor · latent regression | effect gamma: gamma -1.03 | Hurt |
Closest papers by meaning
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.88)
- AI Progress Should Be Measured by Capability-Per-Resource, Not Scale Alone: A Framework for Gradient-Guided Resource Allocation in LLMs (NeurIPS 2025 · similarity 0.87)
- Establishing Best Practices in Building Rigorous Agentic Benchmarks (NeurIPS 2025 · similarity 0.86)
- Can We Trust Item Response Theory for AI Evaluation? (NeurIPS 2026 · similarity 0.86)
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.86)
- The Leaderboard Illusion (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
