Research map / Multi-agent systems and games / LLM agents
Establishing Best Practices in Building Rigorous Agentic Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Agentic Benchmark Checklist (ABC) for outcome validity, task validity, and reporting, applies it to ten agentic benchmarks, and revises CVE-Bench to reduce overestimation by 33%.
Problem
Agentic benchmarks often have task-setup and outcome-grading flaws that under- or overestimate agent performance by up to 100% relative, undermining the validity of reported results.
Approach
Decomposes agentic evaluation validity into outcome validity and task validity and turns them into an actionable checklist (ABC) of checks spanning task setup, outcome grading, and result reporting, which is then used to audit existing benchmarks and improve CVE-Bench.
Key results
- Applying ABC to CVE-Bench reduces performance overestimation by 33% in absolute terms, confirmed by cybersecurity experts.
- A trivial agent achieves a 38% success rate on τ-bench airline by returning empty responses on impossible tasks.
- SWE-Lancer lets an agent score 100% by overwriting test files without solving tasks; KernelBench overestimates correctness by about 31%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the ABC checklist to revise CVE-Bench evaluation | CVE-Bench · CVE-Bench | performance overestimation: 33% | Helped |
| Check for actual SLEEP execution rather than SLEEP text in the database log | CVE-Bench time-based SQL injection evaluation · CVE-Bench time-based SQL injection | performance overestimation: 32.5% | Hurt |
| Deny external requests on the outbound server | CVE-Bench outbound-server evaluation · CVE-Bench outbound server | agent success rate: 10% | Hurt |
| Avoid trivial states or substrings as ground truth | τ-bench · τ-bench | performance overestimation: 38% | Hurt |
| Prevent tasks from being passed by listing every possible answer | τ-bench · τ-bench | performance overestimation: 40% | Hurt |
| Validate LLM-judge accuracy and consistency and prevent substring-match shortcuts | WebArena · WebArena | performance overestimation: 1.4–5.2% | Hurt |
| Isolate agents from benchmark test files | SWE-Lancer · SWE-Lancer | score without solving tasks: 100% | Hurt |
| Vary fuzz-test shapes and memory layouts, not only tensor values | KernelBench · KernelBench | kernel-correctness overestimation: 31% | Hurt |
| Repair website-dependent HTML selectors in the chrome task section | OSWorld UI-TAR evaluation · OSWorld chrome task section | UI-TAR performance underestimation: 28% | Hurt |
Surprising findings
- A trivial agent returning empty responses outperforms a GPT-4o-based agent on τ-bench.
- Among ten assessed benchmarks, 7 have outcome-validity flaws, 7 have task-validity flaws, and all 10 have reporting limitations.
Limitations
- Analysis covers only 17 benchmarks used by top AI providers between January 2024 and March 2025, so the checklist may not be exhaustive.
- Findings reflect benchmark versions at the time of writing and may change in later revisions.
- Taxonomy is grounded in current understanding of agent capabilities and may not cover future failure modes.
Experimental setup
Datasets: SWE-bench-Verified τ-bench WebArena SWE-Lancer KernelBench BIRD Cybench MLE-bench GAIA OSWorld CVE-Bench
Benchmarks: SWE-bench-Verified τ-bench WebArena SWE-Lancer KernelBench OSWorld CVE-Bench
Metrics: success rate performance overestimation/underestimation (relative and abs checklist score
Baselines: UI-TARS GPT-4o-based agent
Builds on: SWE-bench τ-bench WebArena CVE-Bench
Closest papers by meaning
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.90)
- Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.88)
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (ICLR 2026 · similarity 0.88)
- Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability (NeurIPS 2026 · similarity 0.88)
- Agent-as-a-Judge: Evaluate Agents with Agents (ICML 2025 · similarity 0.88)
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.