Research map / Large language models / Agents, code and math

MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

NeurIPS 2026 · Large language models › Agents, code and math · approach: Software engineering agents · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MCP-Atlas, a 1,000-task benchmark run against 36 real MCP servers with claim-level partial-credit scoring and an 11-category failure taxonomy, evaluating 20 frontier models.

Problem

Existing MCP agent evaluations lack realistic multi-server workflows, authentic (non-mock) servers, and reproducible scoring that is not biased by agent verbosity or style.

Approach

Answer-centric claim-level rubric: final answers are scored against atomic factual claims grounded in tool outputs, so alternative valid tool-call trajectories receive credit; prompts avoid naming tools and include plausible distractor tools; an 11-mode diagnostic taxonomy separates tool-call from cognitive failures.

Key results

• Muse Spark reaches 82.2% pass rate at 0.75 claim coverage, followed by Claude Opus 4.7 (79.1%) and Gemini 3.1 Pro Preview (78.2%); open-source GLM-5.1 reaches 75.6%.
• 63.3% of diagnosed failures are cognitive and 36.7% are tool-call related across 20 models.
• o3 Pro reaches only 44.5% because 40.1% of its failed trajectories contain no tool invocation.

Surprising findings

• o3 Pro, a leading reasoning model, finishes near the bottom because it makes no tool calls on 40% of its failed tasks.
• Several high-performing models fail after successful tool execution due to premature stopping or incorrect synthesis.

Limitations

• Absolute pass rates depend on LLM judge choice, with a 2-5 point cross-judge band.
• Live servers and APIs drift over time, limiting reproducibility of trajectories.
• Tasks are English-only and a fixed 100-call budget is used.

Experimental setup

Datasets: MCP-Atlas tasks (1,000 expert-authored tasks)

Benchmarks: MCP-Atlas

Metrics: pass rate at 0.75 claim coverage claim coverage score failure mode share wall-clock time per task

Baselines: MCP-Universe MCPEval MCP-Bench Toolathlon MCPMark LiveMCPBench MCPVerse MCP-RADAR

Builds on: Model Context Protocol MCP-Universe Toolathlon MCP-Bench LiveMCPBench

Closest papers by meaning

• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.91)
• MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools (AAAI 2026 · similarity 0.90)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.89)
• MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use (ICLR 2026 · similarity 0.89)
• The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution (ICLR 2026 · similarity 0.88)
• MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation in Multi-Server MCP Agents (NeurIPS 2026 · similarity 0.88)