Research map / Multi-agent systems and games / LLM agents
Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HAL, a standardized parallel agent-evaluation harness plus multidimensional leaderboard (models × scaffolds × benchmarks, accuracy vs. cost) and LLM-aided log analysis via Docent, running 21,730 rollouts across 9 benchmarks for about $40,000.
Problem
Agent evaluations are slow, non-standardized, cost-opaque, scaffold-confounded, and blind to shortcuts and unsafe behaviors, so reported benchmark scores do not reliably reflect real-world agent performance.
Approach
A unified harness that decouples scaffolds from benchmarks, orchestrates hundreds of parallel Azure VMs, tracks token cost via Weave and LiteLLM, and pairs Pareto accuracy-cost analysis with LLM-based (Docent) rubric analysis of agent logs to surface shortcuts and reliability failures.
Key results
- 21,730 agent rollouts across 9 benchmarks and 9 models cost about $40,000, with the harness cutting evaluation time from weeks to hours via hundreds of parallel VMs.
- For 21 of 36 model-scaffold-benchmark runs comparing reasoning effort settings, higher reasoning effort did not improve accuracy.
- Gemini 2.0 Flash lies on the accuracy-cost Pareto frontier in 7 of 9 benchmarks, while Claude Opus 4.1 appears on the cost frontier only once.
Surprising findings
- Higher reasoning effort reduced or failed to improve accuracy in most comparisons (21 of 36 runs).
- A widely used TAU-bench few-shot scaffold leaked test-set examples into its demonstrations, invalidating its results, and was only caught by log analysis.
Limitations
- Single runs for most configurations due to high cost, so no statistical uncertainty estimates.
- Public test sets used for GAIA and AssistantBench; SWE-bench Verified Mini (50 tasks) instead of full set; original TAU-bench rather than tau2.
- Incomplete evaluation matrix (142 of 186 runs reported); Claude Opus 4.1 not run on Online Mind2Web due to cost; latency measurements unreliable under parallel execution.
Experimental setup
Datasets: AssistantBench CORE-Bench Hard GAIA Online Mind2Web SciCode ScienceAgentBench SWE-bench Verified Mini USACO TAU-bench Airline
Benchmarks: AssistantBench CORE-Bench Hard GAIA Online Mind2Web SciCode ScienceAgentBench SWE-bench Verified Mini USACO TAU-bench Airline
Metrics: accuracy dollar cost token cost pareto frontier
Baselines: AgentBench AgentBoard BrowserGym AgentGym Galileo Leaderboard AISI Inspect
Builds on: HELM LM Evaluation Harness Docent Weave LiteLLM
Closest papers by meaning
- Agent-as-a-Judge: Evaluate Agents with Agents (ICML 2025 · similarity 0.90)
- General Agent Evaluation (NeurIPS 2026 · similarity 0.88)
- FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics (NeurIPS 2026 · similarity 0.88)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.88)
- Establishing Best Practices in Building Rigorous Agentic Benchmarks (NeurIPS 2025 · similarity 0.88)
- Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.