Research map / Multi-agent systems and games / LLM agents
AgentSuite: Toward More Reliable Agent Evaluation with a Component-Based Benchmark Auditing Pipeline
Hyewon Suh, Binfei Ji, Seojune Lee, Rishi Khare, Basit Khan, Hyunjun Kim, Tianyi Zhang, Venkat Krishna Srinivasan, Peter Belcak, Shizhe Diao, Pavlo Molchanov, Yingyan Celine Lin and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COBA, a component-based auditing pipeline that checks agent benchmarks for hidden flaws across instructions, environments, tools, ground-truth trajectories, and evaluation protocols.
Problem
LLM agent benchmarks often contain hidden flaws from interactions among instructions, environments, tools, ground-truth trajectories, and evaluation protocols, which confound model errors with benchmark artifacts and undermine leaderboard comparisons.
Approach
A component-based auditing pipeline (COBA) that decomposes an agent benchmark into its constituent components and audits each, including their interactions, to surface benchmark validity flaws before trusting agent evaluation results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use taxonomy-guided prompt instead of generic critique prompt (ablation) | COBA judge · ACEBench disagreement cases | win rate on disagreements: 72.8% | Improved |
| use taxonomy-guided prompt instead of generic critique prompt (ablation) | COBA judge · CFB (ComplexFuncBench) disagreement cases | win rate on disagreements: 73.3% | Improved |
| use taxonomy-guided prompt instead of generic critique prompt (ablation) | COBA judge · ACEBench and CFB | number of flagged issues: generic flags 1.7x more issues | Improved |
| remove flawed tasks via COBA filtering vs random removal of same number | ACEBench ranking · ACEBench | Kendall tau: 0.871 vs 0.960 | Improved |
| remove flawed tasks via COBA filtering vs random removal of same number | ACEBench ranking · ACEBench | pairwise flip rate: 5.75% vs 1.50% | Improved |
| remove inflation-prone flawed tasks | ACEBench · ACEBench | average model performance: 1.000 -> 0.780 | Improved |
| remove deflation-dominated flawed tasks | ACEBench · ACEBench | average model performance: 0.555 -> 0.780 | Improved |
| filter flagged tasks before evaluation | gpt-4o-20240806 · ACEBench | score: 85.2 -> 88.3 | Improved |
| filter flagged tasks before evaluation | gpt-oss-120b · ACEBench | rank: 11 -> 16 | Hurt |
| filter flagged tasks before evaluation | claude-4.5-sonnet-thinking-on-10k · ACEBench | rank: 6 -> 3 | Improved |
Closest papers by meaning
- Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis (ICLR 2026 · similarity 0.90)
- PACE: A Proxy for Agentic Capability Evaluation (NeurIPS 2026 · similarity 0.90)
- PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents (NeurIPS 2026 · similarity 0.89)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.89)
- Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.88)
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
