Research map / Multi-agent systems and games / LLM agents

OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OSWorld-MCP, a benchmark extending OSWorld with 158 curated MCP tools so computer-use agents can be evaluated on GUI operation, tool invocation, and decision-making jointly.

Problem

Computer-use agent benchmarks evaluate only GUI actions and ignore MCP tool invocation, making comparisons between tool-using and GUI-only agents unfair and leaving tool-use decision-making unmeasured.

Approach

Combine an o3-driven automated code-generation pipeline (generate code solutions, filter, wrap into MCP tools) with manually curated tools from existing MCP servers, then let agents choose at each step between GUI actions and 158 MCP tools, measured by accuracy, Tool Invocation Rate (TIR) and Average Completion Steps (ACS).

Key results

• OpenAI o3 accuracy at 15 steps rises from 8.3% (GUI-only) to 20.4% with MCP tools.
• Claude 4 Sonnet reaches 43.3% at 50 steps with MCP vs 40.1% GUI-only; its TIR is 36.3%, the highest among end-to-end models.
• Gemini-2.5-Pro overall accuracy at 15 steps rises from 7.4% to 20.5% with MCP tools; removing RAG tool filtering drops it to 15.5%.

Surprising findings

• Tool descriptions presented in a random order raised Gemini-2.5-Pro accuracy from 20.5% to 22.7%, showing sensitivity to tool ordering in the prompt.
• Combining multiple MCP tools is harder for LMMs than combining GUI actions; efficacy of tools declines in tasks requiring tool combinations.

Limitations

• Low tool invocation rates for all evaluated models, with the best at 36.3%.
• Agents struggle to compose multiple tools and to select the correct tool from a full list of 158.
• MCP servers not developed for GIMP and Thunderbird due to OSWorld version constraints.

Experimental setup

Datasets: OSWorld MCP servers (existing open-source)

Benchmarks: OSWorld-MCP OSWorld

Metrics: task accuracy tool invocation rate average completion steps

Baselines: Agent-S2.5 GUI-only OSWorld configuration

Builds on: OSWorld CoAct Agent-S2.5 Model Context Protocol

Closest papers by meaning

• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.92)
• MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools (AAAI 2026 · similarity 0.91)
• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (NeurIPS 2026 · similarity 0.91)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.91)
• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.91)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.89)