Research map / Multi-agent systems and games / LLM agents

MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MCPMark, a 127-task benchmark of CRUD-diverse MCP workflows across five servers (Notion, GitHub, Filesystem, PostgreSQL, Playwright) with programmatic verification, and evaluates frontier LLMs in a minimal tool-calling agent showing low pass^4 scores.

Problem

Existing MCP benchmarks are narrow, focusing on read-heavy or shallow-interaction tasks, and do not capture the complexity of realistic multi-step workflows or allow reliable automated verification.

Approach

Tasks are built through a human-AI collaborative pipeline on curated initial environment states, each paired with a programmatic verification script and state reset, emphasizing diverse CRUD operations and long interaction horizons; reliability is measured with pass^4 in addition to pass@1 and pass@4.

Key results

• gpt-5-medium reaches 52.56% pass@1 and 33.86% pass^4, the best among evaluated models.
• claude-sonnet-4 and o3 fall below 30% pass@1 and 15% pass^4.
• Models average 16.2 turns and 17.4 tool calls per task.

Surprising findings

• More turns do not necessarily yield better performance; kimi-k2-instruct overcalls with diminishing success.
• Higher cost does not imply higher accuracy across runs.

Limitations

• Task creation pipeline is labor-intensive (3-5 hours per task) and hard to scale.
• Steep difficulty limits usefulness for evaluating smaller models (<=100B excluded).
• Minimal agent framework omits optimizations, so scores may underestimate achievable performance.

Experimental setup

Datasets: MCPMark tasks (127)

Benchmarks: MCPMark

Metrics: pass@1 pass@4 pass^4 turns tool calls cost

Baselines: MCP-Universe LiveMCP-101 LiveMCPBench MCPEval MCP-AgentBench

Builds on: Model Context Protocol WebArena ReAct LiteLLM

Closest papers by meaning

• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.90)
• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (NeurIPS 2026 · similarity 0.89)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.89)
• Evaluating Test-Time Scaling of General LLM Agents (NeurIPS 2026 · similarity 0.88)
• MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools (AAAI 2026 · similarity 0.88)
• MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation (ICML 2026 · similarity 0.87)