Research map / Vision-language and multimodal / Human and embodied modeling
AGI-Elo: How Far Are We From Mastering A Task?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AGI-Elo, a Glicko-style rating system that jointly rates test-case difficulty and model/human competency across six vision, language, and action benchmarks to quantify gaps to task mastery.
Problem
Existing AI benchmark metrics are aggregate and model-centric, obscuring per-test-case difficulty distributions and the remaining gap between current models and full mastery of a task.
Approach
Treat test cases and agents as heterogeneous players in inter-category matches (IRT-like, Elo/Glicko updates with a score mapping from task metrics), so ratings for both difficulty and competency are jointly estimated and used to predict per-case performance and oracle competency gaps.
Key results
- Highest-rated AI models show competency gaps of about 233–387 rating points to 50% oracle mastery across six datasets, and 1031–1185 to 99% oracle mastery.
- A human expert on NAVSIM reaches near-oracle competency with a gap of only 20.7 rating points at 50% confidence.
- Rating-based predictions show high consistency with original metrics (|rho_a| ≈ 0.998–1.000) with low MAE (0.017–0.066) across datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AGI-Elo ratings to predict agent performance | AGI-Elo · ImageNet val | MAE: 0.0476 | Helped |
| Use AGI-Elo ratings to predict agent performance | AGI-Elo · COCO val | MAE: 0.0167 | Helped |
| Use AGI-Elo ratings to predict agent performance | AGI-Elo · MMLU test | MAE: 0.0662 | Helped |
| Use AGI-Elo ratings to predict agent performance | AGI-Elo · LiveCodeBench test | MAE: 0.0446 | Helped |
| Use AGI-Elo ratings to predict agent performance | AGI-Elo · Waymo val | MAE: 0.0354 | Helped |
| Use AGI-Elo ratings to predict agent performance | AGI-Elo · NAVSIM test | MAE: 0.0546 | Helped |
| Estimate agent competency using AGI-Elo | AI models across six datasets · Six datasets | competency gap @50%: 233–387 rating points | Hurt |
Surprising findings
- A human expert on NAVSIM is near oracle-level under PDM score, the only agent close to mastery.
Limitations
- Limited computational resources constrain the experimental scale.
- Selected datasets and models may not represent SOTA performance fully.
- Relies on the assumption that transitivity holds in heterogeneous agent-test case matches and that the dataset is a faithful miniature of the real distribution.
Experimental setup
Datasets: ImageNet COCO MMLU LiveCodeBench Waymo Open Motion Dataset NAVSIM
Benchmarks: ImageNet COCO MMLU LiveCodeBench Waymo Open Motion Dataset NAVSIM
Metrics: Acc@1 AP@[.50:.90] accuracy PassAll mAP PDMS Spearman rank correlation MAE MSE
Baselines: Elo Glicko Chatbot Arena
Builds on: Elo Glicko Item Response Theory
Closest papers by meaning
- Position: Levels of AGI for Operationalizing Progress on the Path to AGI (ICML 2024 · similarity 0.88)
- GAIA: a benchmark for General AI Assistants (ICLR 2024 · similarity 0.86)
- Position: Stop Making Unscientific AGI Performance Claims (ICML 2024 · similarity 0.86)
- Position: AI Evaluation Should Learn from How We Test Humans (ICML 2025 · similarity 0.86)
- The AI Race: Why Current Neural Network-based Architectures are a Poor Basis for Artificial General Intelligence (AAAI 2025 · similarity 0.85)
- The Alignment Problem from a Deep Learning Perspective (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.