Research map / Multi-agent systems and games / LLM agents
MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents
Yizhen Jiang, 浩伟 郭, Tianyi Bai, Zengjie Hu, Lu Ma, Zhengyang Zhao, Haoze Sun, Peng Pei, Wentao Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MCPHallu, a benchmark for diagnosing reasoning, execution, and memory hallucinations in LLM agents that use the Model Context Protocol to call tools across servers.
Problem
Existing MCP agent benchmarks report aggregate task success rates and do not attribute failures to specific reasoning, execution, or memory breakdowns.
Approach
A benchmark that categorizes MCP agent failures into reasoning, execution, and memory hallucinations so that failures can be attributed to specific breakdown types rather than only aggregate success rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Let the agent issue four or more tool calls on an infeasible goal instead of stopping early | MCP agents · MCPHallu Unreachable Goal | pass rate: 65.0% (0 calls) -> 27.9% (1-3) -> 0.0% (4+) | Hurt |
| Mix confusion tools with the correct tool | MCP agents · MCPHallu Tool Misuse | pass rate: 84.7% -> 32.3% | Hurt |
| Use a plan-and-execute scaffold and infeasibility-aware prompt (ablation) | MCP agents · MCPHallu | hardest subtypes | No clear effect |
| Use deterministic components only instead of full score including LLM judge (ablation) | Scoring protocol · MCPHallu model ranking | Spearman rho vs full score: rho 0.97, no model moves more than 2 ranks | No clear effect |
| Swap LLM judge GPT-4o for Claude Sonnet 4.5 (ablation) | Scoring protocol · MCPHallu | score: change at most 0.016 on average, r = 0.914 | No clear effect |
| Remove the 19% mixed/other failure tasks (ablation) | Benchmark · MCPHallu | pairwise model ranking | No clear effect |
| Increase task difficulty from Easy to Hard | MCP agents · MCPHallu | score | Hurt |
Experimental setup
Benchmarks: MCPHallu
Metrics: task success rate hallucination attribution
Builds on: Model Context Protocol
Closest papers by meaning
- MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools (AAAI 2026 · similarity 0.91)
- MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (NeurIPS 2026 · similarity 0.90)
- OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.89)
- MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use (ICLR 2026 · similarity 0.89)
- EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.88)
- AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
