Research map / Multi-agent systems and games / LLM agents
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OSWorld, a scalable real-computer environment (VMs on Ubuntu/Windows/macOS) and a 369-task execution-evaluated benchmark for multimodal computer-use agents, showing best LLM/VLM agents reach only ~12% vs 72% for humans.
Problem
Existing computer-use benchmarks either lack executable environments or are confined to specific apps or domains, so they cannot evaluate open-ended, cross-application computer tasks reliably.
Approach
A VM-based real desktop environment with raw keyboard/mouse (pyautogui) action space, intermediate initial-state setup configs, and example-specific execution-based evaluation scripts, used to build an open-ended multi-app benchmark.
Key results
- Best agent (GPT-4V with accessibility tree) reaches 12.24% overall success (GPT-4 a11y tree 12.24%, GPT-4V SoM 11.77%) versus 72.36% human accuracy.
- Screenshot-only GPT-4V and Gemini-Pro-vision achieve only 5.26% and 5.80%.
- Multi-app workflow tasks reach at most 6.57% success for the best baseline, versus 13.74% on single-app tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use screenshot plus an accessibility tree instead of screenshot alone | GPT-4V · OSWorld overall | success rate: 5.26% to 12.17% | Helped |
| Use Set-of-Marks instead of screenshot plus accessibility tree (ablation) | GPT-4V · OSWorld overall | success rate: 12.17% -> 11.77% | Hurt |
| Add more text-based trajectory history (ablation) | GPT-4V (Set-of-Marks) · OSWorld subset (10%) | success rate | Helped |
| Add trajectory history to screenshot-only input (ablation) | GPT-4V (Screenshot) · OSWorld subset (10%) | success rate | No clear effect |
| Increase screenshot resolution (ablation) | GPT-4V (Screenshot) · OSWorld subset (10%) | success rate | Helped |
| Downsample SoM screenshots to 768×432 (ratio 0.4) (ablation) | GPT-4V (Set-of-Marks) · OSWorld subset (10%) | success rate | Helped |
| Further reduce SoM screenshot resolution to down-sampling ratio 0.2 (ablation) | GPT-4V (Set-of-Marks) · OSWorld subset (10%) | success rate | Hurt |
| Perturb application window position or size, or clutter the screen with irrelevant windows (ablation) | GPT-4V (Set-of-Marks) · OSWorld subset of 28 tasks | success rate: 50.79% to over 60% to even 80% performance drop | Hurt |
| Use GPT-4V instead of Claude-3 Opus | Screenshot-based agents · OSWorld | accuracy: 2.84% to 7.76% | Helped |
Surprising findings
- Claude-3 Opus, competitive on common benchmarks, falls far behind GPT-4V as a computer agent.
- Set-of-Mark prompting reduced GPT-4V performance relative to direct screenshot plus a11y input in some comparisons, attributed to high-resolution noisy screens.
Limitations
- Evaluation scripts may have false positives/negatives despite quality control
- Windows and macOS coverage limited; Windows tasks are only a small analytic set and macOS excluded
- No assessment of unsafe side effects of agents
Experimental setup
Datasets: NL2Bash Mind2Web SheetCopilot PPTC GAIA
Benchmarks: OSWorld
Metrics: success rate human completion time
Baselines: Mixtral-8x7B Llama-3-70B GPT-3.5 GPT-4 Gemini-Pro Gemini-Pro-1.5 Qwen-Max GPT-4o CogAgent GPT-4V Gemini-ProV Claude-3-Opus
Builds on: WebArena VisualWebArena Set-of-Mark pyautogui
Closest papers by meaning
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.93)
- OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.92)
- Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.91)
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (ICLR 2025 · similarity 0.89)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.89)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.