Research map / Multi-agent systems and games / LLM agents
Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Windows Agent Arena, a scalable benchmark of 154 Windows OS tasks with execution-based evaluation parallelized on Azure, plus the OmniParser-based GPT-4V agent NAVI reaching 19.5% success versus 74.5% for humans.
Problem
Existing computer-use agent benchmarks are either modality/domain-limited or too slow to evaluate (days per full run), and none provide a realistic, scalable real-OS environment for Windows, the most common desktop OS.
Approach
Adapt the OSWorld framework to a real Windows 11 VM running in Docker containers, with per-task JSON configs and execution-based evaluators, and parallelize one worker per task on Azure ML so a full run takes about 20 minutes; a baseline agent (NAVI) combines UIA-tree and pixel-based Set-of-Marks (OmniParser) with a VLM.
Key results
- Best NAVI configuration (UIA + OmniParser SoM with GPT-4V-1106) reaches 19.5% success on WAA versus 74.5% for unassisted humans.
- Adding UIA markers to OmniParser boosts GPT-4V-1106 performance by 57% relative, per the paper's analysis.
- On Mind2Web, NAVI with Image+SoM (DOM + Pixel) and GPT-4o reaches 45.2% step success rate versus 38.3% for SeeAct (GPT-4V).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add UIA markers to pixel-based Omniparser (ablation) | Navi with GPT-4V-1106 · Windows Agent Arena | success rate: 12.5% -> 19.5% | Helped |
| add image input to SoM (DOM + Pixel) (ablation) | Navi with GPT-4o · Mind2Web | Step Success Rate: 35.1% -> 45.2% | Helped |
| add pixel SoM to Image + SoM (DOM) (ablation) | Navi with GPT-4o · Mind2Web | Step Success Rate: 40.7% -> 45.2% | Helped |
| replace text-only DOM input with Image+SoM (DOM + Pixel) (ablation) | Navi with GPT-4o · Mind2Web | Step Success Rate: 24.6% -> 45.2% | Helped |
| use Image+SoM (DOM + Pixel) instead of Image+SoM (DOM) (ablation) | Navi with GPT-4o · Mind2Web | Operation F1: 84.3% -> 85.8% | Helped |
| use Image+SoM (DOM + Pixel) instead of SoM (Pixel) (ablation) | Navi with GPT-4o · Mind2Web | Element Accuracy: 40.6% -> 47.3% | Helped |
| use Image+SoM (DOM + Pixel) instead of the SeeAct configuration | Navi versus SeeAct · Mind2Web | Operation F1: 71.8% -> 85.8% | Helped |
| use the strongest Navi configuration instead of the text-only DOM configuration (ablation) | Navi with GPT-4o · Mind2Web | Element Accuracy: 26.3% -> 47.3% | Helped |
Surprising findings
- Adding high-quality UIA markers to pixel-based detectors does not uniformly help; gains vary by detector and model.
- Phi3-V hallucinates when planning multiple steps or given long contexts, and GPT-4o performs worse than GPT-4V-1106 in the best UIA+OmniParser setting.
Limitations
- Absolute agent success remains low (19.5%) compared with humans (74.5%).
- Windows 11 VM image cannot be redistributed due to licensing and must be downloaded by users.
- Reward is binary at task end with no intermediate partial credit.
Experimental setup
Datasets: OSWorld task configurations (adapted) Mind2Web
Benchmarks: Windows Agent Arena Mind2Web
Metrics: task success rate element accuracy operation F1 step success rate
Baselines: GPT-4o GPT-4o-mini Phi3-V GPT-4V-1106 SeeAct
Builds on: OSWorld Set-of-Marks chain-of-thought
Closest papers by meaning
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.93)
- OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.91)
- Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.91)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.90)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.90)
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.