Research map / Multi-agent systems and games / LLM agents

MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MCP-AgentBench, a 600-query benchmark over a 33-server, 188-tool MCP testbed, with an outcome-oriented LLM-as-judge (MCP-Eval) that scores 10 LLMs in ReAct and native tool-calling modes.

Problem

Existing tool-use benchmarks such as BFCL do not capture agent performance in MCP-mediated tool interactions, so their scores can misrank models relative to real-world use.

Approach

A stateless, text-based MCP server testbed with queries generated along two complexity axes (server scope x call dependency, giving six categories), reference answers produced via LLM trajectories plus expert review, and pass/fail judgment on final-answer outcomes rather than execution paths.

Key results

• Qwen3-235B-A22B with ReAct achieved the highest overall pass rate (64.7%), while Kimi K2 with TC reached 61.0% and Claude 4 Sonnet with TC reached 58.0%.
• GPT-4o scored lowest among proprietary models (27.8% ReAct, 30.7% TC).
• MCP-Eval agreed with human majority vote on 91.67% of 60 items (Cohen's kappa 0.734).

Surprising findings

• Qwen3-235B-A22B leads under ReAct but collapses to 40.2% under native tool calling, often failing to emit a required tool call.
• Claude 4 Sonnet improves on harder multi-server tasks, which the authors attribute to reduced reliance on parametric knowledge when complexity forces tool use.

Limitations

• Only stateless, text-based servers were included, excluding stateful and non-text modalities.
• Judge is a single LLM (o3-mini-high) validated on only 60 items.
• Reference answers are time-sensitive and reflect outcomes at annotation time.

Experimental setup

Datasets: MCP-AgentBench queries (600, LLM-generated and human-verified)

Benchmarks: MCP-AgentBench

Metrics: pass rate token consumption percentage agreement Cohen's kappa Fleiss' kappa

Baselines: BFCL MCP-Bench MCPEval MCP-RADAR ToolBench API-Bank

Builds on: ReAct BFCL MCP-Bench MCPEval

Closest papers by meaning

• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.91)
• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (NeurIPS 2026 · similarity 0.90)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.89)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.89)
• Harbor Adapters and Harbor-Mix: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation (NeurIPS 2026 · similarity 0.89)
• AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.88)