Research map / Large language models / Agents, code and math
Benchmarking at the Edge of Comprehension
Samuele Marro, Jialin Yu, Emanuele La Malfa, Oishi Deb, Jiawei Li, Yibo Yang, Ebey Abraham, Sunando Sengupta, Eric Sommerlade, Michael J. Wooldridge, Philip Torr
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use critique-resilient answerer Elo as a model score | Critique-Resilient Benchmarking · AIME 2025, BRUMO 2025, HMMT Feb 2025 | Spearman rho: 0.851, 0.830, 0.819 correlation with external benchmarks | Improved |
| fit Bradley-Terry scores versus a base-rate predictor on held-out questions | itemized bipartite Bradley-Terry model · 5-fold validation | accuracy: 0.646 -> 0.922 | Improved |
| replace human adjudicators with weak models (GPT-3.5, GPT-4o) (ablation) | adjudication stage · agreement with human verdicts | Spearman rho of rankings: rho in [0.98, 1], tau in [0.93, 1] | No clear effect |
| vary prompt design (three-way prompt ablation) (ablation) | answerer rankings · prompt sensitivity | Kendall W: Kendall W = 0.952 | No clear effect |
| stronger models as pair members (frontier vs GPT-3.5 pairs) on mutual incompatibility | non-proven-wrong answers · answer-pair compatibility | incompatible pairs (%): 14-18% (GPT-3.5 pairs) -> 0-0.4% (frontier pairs) | Improved |
| compare critique agreement to answer correctness across models | critics · GPT-5.4-high as ground truth proxy | agreement rate: critique agreement 0.88-0.96 vs answer correctness 0.16-0.99 | No clear effect |
Closest papers by meaning
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026 · similarity 0.89)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.88)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.88)
- \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.88)
- Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.87)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
