AI Paper Atlas

Research map / Multi-agent systems and games / LLM agents

MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents

Yizhen Jiang, 浩伟 郭, Tianyi Bai, Zengjie Hu, Lu Ma, Zhengyang Zhao, Haoze Sun, Peng Pei, Wentao Zhang

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MCPHallu, a benchmark for diagnosing reasoning, execution, and memory hallucinations in LLM agents that use the Model Context Protocol to call tools across servers.

Problem

Existing MCP agent benchmarks report aggregate task success rates and do not attribute failures to specific reasoning, execution, or memory breakdowns.

Approach

A benchmark that categorizes MCP agent failures into reasoning, execution, and memory hallucinations so that failures can be attributed to specific breakdown types rather than only aggregate success rates.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Let the agent issue four or more tool calls on an infeasible goal instead of stopping earlyMCP agents · MCPHallu Unreachable Goalpass rate: 65.0% (0 calls) -> 27.9% (1-3) -> 0.0% (4+)Hurt
Mix confusion tools with the correct toolMCP agents · MCPHallu Tool Misusepass rate: 84.7% -> 32.3%Hurt
Use a plan-and-execute scaffold and infeasibility-aware prompt (ablation)MCP agents · MCPHalluhardest subtypesNo clear effect
Use deterministic components only instead of full score including LLM judge (ablation)Scoring protocol · MCPHallu model rankingSpearman rho vs full score: rho 0.97, no model moves more than 2 ranksNo clear effect
Swap LLM judge GPT-4o for Claude Sonnet 4.5 (ablation)Scoring protocol · MCPHalluscore: change at most 0.016 on average, r = 0.914No clear effect
Remove the 19% mixed/other failure tasks (ablation)Benchmark · MCPHallupairwise model rankingNo clear effect
Increase task difficulty from Easy to HardMCP agents · MCPHalluscoreHurt

Experimental setup

Benchmarks: MCPHallu

Metrics: task success rate hallucination attribution

Builds on: Model Context Protocol

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy