Research map / Multi-agent systems and games / LLM agents
General Agent Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Unified Protocol and the Exgentic harness to run unmodified general agents (ReAct, Smolagent, OpenAI Solo, Claude Code, ReAct Short) on six benchmarks across five backbone LLMs, releasing an Open General Agent Leaderboard of 25 configurations.
Problem
No systematic study measures how agent architecture shapes performance when the same general-purpose agent is evaluated across heterogeneous protocols and unfamiliar benchmarks, because existing harnesses require per-benchmark wiring or fixed protocols and benchmarks assume human-authored prompts.
Approach
A task/context/actions "narrow waist" protocol mediates between native agent interfaces (tool-calling, MCP, code, CLI) and benchmarks, enabling a full factorial study of agent architecture x backbone model x benchmark without modifying either side.
Key results
- Best configuration (OpenAI Solo + Claude Opus 4.5) reaches 0.73 bench-weighted success; model choice explains 27.8% of variance versus 0.5% for architecture among closed-source configs.
- Architecture spread within a model reaches 12pp for closed-source and 14-18pp for open-weight backbones; Kimi-K2.5 with autonomous architectures scores 0.00 on tau2-Telecom vs 0.83 with ReAct.
- On 4 of 6 benchmarks the best general-agent configuration is statistically indistinguishable from top domain-specific scores.
Surprising findings
- Open-weight models show 'generality sinks': 94% of Kimi-K2.5 sessions with autonomous architectures on tau2-Bench take zero steps due to first-turn protocol violations.
- GPT 5.2 API's 128-tool limit causes 0.00 scores on AppWorld without tool shortlisting.
Limitations
- Only five LLMs, five agents and six benchmarks with 100 tasks each, leading to Wilson CI half-widths of about 7-13pp per cell.
- Open-weight sink findings tied to two checkpoints; generalization untested.
- Multimodal and continuous-action extensions not evaluated.
Experimental setup
Datasets: SWE-Bench Verified tau2-Bench AppWorld BrowseComp+
Benchmarks: SWE-Bench Verified BrowseComp+ tau2-Bench Airline tau2-Bench Retail tau2-Bench Telecom AppWorld
Metrics: success rate cost per task average steps bench-weighted mean cost-efficiency eta-squared variance decomposition
Baselines: ReAct Smolagent OpenAI Agents SDK solo Claude Code HAL Inspect
Builds on: ReAct Model Context Protocol mini-SWE-agent Harbor BrowserGym
Closest papers by meaning
- ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.88)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.88)
- Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.88)
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026 · similarity 0.88)
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.88)
- Harbor Adapters and Harbor-Mix: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.