Research map / Multi-agent systems and games / LLM agents
MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MCP-Persona, a benchmark of 173 human-verified personalized tasks over simulated MCP servers (Lark, Rednote, Slack, Notion, etc.), using code-based tool simulation (Tool-Traverse) and tree-structured user contexts, and finds SOTA agents score under 50%.
Problem
Existing tool-use benchmarks focus on generic information-seeking tools and synthetic environments, failing to evaluate LLM agents on personalized MCP tools tied to individual accounts and local data, which are hard to deploy and share due to privacy and account binding.
Approach
Simulate stateful personal MCP servers by traversing authentic function calls (seed FCs plus adversarial failure induction) and having an LLM synthesize executable Python simulators over a user-rooted context tree, then generate personalized tasks from tool chains with implicit-context fuzzification and human verification.
Key results
- Claude-Sonnet-4.5 achieves the best overall Acc of 38.66 on MCP-Persona, and no model exceeds 50% accuracy.
- Tool-Traverse simulation reaches 94.0% behavioral alignment F1 93.8 on Lark versus 53.3 F1 for the documentation-only Vanilla baseline.
- Restricting candidate tools to ground-truth servers improves some models, e.g., GPT-5 Acc rises from 41.04 to 40.46 only marginally while Gemini-2.5-Pro rises from 20.23 to 22.54.
Surprising findings
- GPT-5 is the most cost-effective model, reaching 36.99 accuracy at about $0.09 per task, with no clear correlation between spending and performance.
- Adding five distractor paragraphs to context degrades some models, though it slightly improves others (e.g., Gemini-2.5-Pro Acc rises from 20.23 to 20.81).
Limitations
- Benchmark has only 173 tasks, and some tools are simulated rather than live, so fidelity is validated only on Lark with 50 traces.
- LLM judge (GPT-4o) misaligns with humans on 8.45% of checkpoints, sometimes over-penalizing alternative tool use.
Experimental setup
Datasets: DBpedia entity descriptions
Benchmarks: MCP-Persona
Metrics: Acc SR-0.8 Exec-Acc F1 TF-IDF ROUGE BLEU METEOR
Baselines: Claude-Sonnet-4.5 GPT-5 Claude-Opus-4.1 o4-mini o3 GPT-4o Grok-4 Gemini-3-Pro Gemini-2.5-Pro Qwen3-Max Qwen3-235B-A22B DeepSeek-V3
Builds on: Self-Instruct MCP-Flow Tau-Bench TOOLATHLON AppWorld
Closest papers by meaning
- MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use (ICLR 2026 · similarity 0.87)
- MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools (AAAI 2026 · similarity 0.87)
- LLM Generated Persona is a Promise with a Catch (NeurIPS 2025 · similarity 0.86)
- OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.86)
- MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (NeurIPS 2026 · similarity 0.85)
- Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.