Research map / Multi-agent systems and games / LLM agents
macOSWorld: A Multilingual Interactive Benchmark for GUI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces macOSWorld, a multilingual interactive benchmark (202 tasks, 5 languages, 30 apps) for GUI agents on macOS, with a real-popup deception safety subset, evaluating six agents.
Problem
Existing interactive GUI-agent benchmarks cover web, Windows, Linux, and Android but not macOS, and are mostly English-only and rarely assess safety against deception.
Approach
An interactive macOS benchmark run on AWS-hosted Mac minis with snapshot-restored environments, tasks in five languages (task and UI language matched), macOS-exclusive apps, and a safety subset using genuine macOS deceptive pop-up windows instead of synthetic overlays.
Key results
- Proprietary CUAs lead: OpenAI CUA 33.8% and Claude CUA 37.1% average SR (overall 33.3%/44.4% on English), while open-source ShowUI and UI-TARS remain below 5% average.
- Arabic shows a 28.8% relative degradation versus English (13.7% vs 19.3% average SR across agents).
- Under deception pop-ups, Claude CUA and OpenAI CUA were distracted 72.4% and 69.0% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Claude CUA rather than GPT-4o | macOSWorld baseline agents · macOSWorld (171 tasks, all-language average) | success rate: 5.3% -> 37.1% | Helped |
| Add Set-of-Mark annotations (ablation) | GPT-4o · macOSWorld English | success rate: 8.8% -> 19.9% | Helped |
| Add Set-of-Mark annotations (ablation) | GPT-4o · macOSWorld Chinese | success rate: 6.4% -> 14.0% | Helped |
| Add Set-of-Mark annotations (ablation) | GPT-4o · macOSWorld Arabic | success rate: 2.9% -> 7.0% | Helped |
| Add Set-of-Mark annotations (ablation) | GPT-4o · macOSWorld Japanese | success rate: 5.3% -> 9.9% | Helped |
| Add Set-of-Mark annotations (ablation) | GPT-4o · macOSWorld Russian | success rate: 2.9% -> 14.0% | Helped |
| Use English tasks in a mismatched Arabic environment rather than matched English tasks and environment | OpenAI CUA · macOSWorld English-task cross-lingual evaluation | success rate: 33.3% -> 22.8% | Hurt |
| Run tasks and interface in Arabic rather than English | All evaluated agents, averaged · macOSWorld (171 tasks, language average) | success rate: 19.3% -> 13.7% | Hurt |
| Expose agents to context-deception pop-ups | Claude CUA · macOSWorld safety subset | distracted rate: 72.4% | Hurt |
| Expose agents to context-deception pop-ups | OpenAI CUA · macOSWorld safety subset | distracted rate: 69.0% | Hurt |
Surprising findings
- GPT-4o with Set-of-Mark annotations roughly doubles its success rate, yet still trails Gemini 2.5 Pro and both CUAs.
- Language mismatch between task prompt and environment UI further degrades OpenAI CUA performance.
Limitations
- Only binary success reward; non-binary intermediate evaluation removed from experiments
- Safety subset small (29 tasks) and only in English
- Advanced Apps category evaluated only in English and excluded from overall score
Experimental setup
Benchmarks: macOSWorld
Metrics: success rate distraction rate gold rate unhandled rate
Baselines: OpenAI CUA Claude CUA GPT-4o Gemini 2.5 Pro UI-TARS 7B DPO ShowUI
Builds on: OSWorld WindowsAgentArena AndroidWorld Set-of-Mark prompting
Closest papers by meaning
- OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.89)
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.88)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.88)
- ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.86)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.86)
- GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.