Research map / Large language models / Agents, code and math
OpenCUA: Open Foundations for Computer-Use Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OpenCUA, an open framework with a cross-OS annotation tool, the AgentNet desktop trajectory dataset (22.6K tasks), reflective long-CoT synthesis, and SFT-trained open CUA models reaching 45.0% on OSWorld-Verified.
Problem
Open-source computer-use agents lack scalable data collection infrastructure, large diverse desktop trajectory datasets, and transparent training recipes, limiting replication and study of CUA capabilities.
Approach
Collect cross-OS human desktop demonstrations with a background recording tool, compress them into state-action pairs, and augment each step with reflective long chain-of-thought (reflector, generator, summarizer pipeline) before SFT of VLMs, plus an offline benchmark (AgentNetBench) with multiple valid gold actions per step.
Key results
- OpenCUA-72B achieves 45.0% average success rate at 100 steps on OSWorld-Verified, the best among open-source models.
- OpenCUA-72B reaches 60.8% on ScreenSpot-Pro and 37.3% on UI-Vision, reported as SOTA on UI-Vision.
- Reflective long CoT raised an ablation performance from 11.5 to 15.3 on OSWorld for Qwen2-VL-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reflective long CoT reasoning (ablation) | Qwen2-VL-7B trained on 14K Win&Mac and 3K Ubuntu trajectories · OSWorld | success rate: 11.5 to 15.3 | Helped |
| train with only L2 CoT instead of a mixture of L1, L2, and L3 (ablation) | OpenCUA training recipe · OSWorld | success rate: 13.1 | Hurt |
| add general-domain text data to agent training (ablation) | Qwen2-VL-7B with grounding warm-up · OSWorld | agentic performance | Helped |
| use multiple visual history screenshots instead of a single screenshot (ablation) | Qwen2-VL-7B trained on 7K trajectories · OSWorld | success rate | Helped |
| increase visual history from 3 to 5 screenshots (ablation) | Qwen2-VL-7B trained on 7K trajectories · OSWorld | success rate | No clear effect |
| use L2 rather than L1 or L3 at inference (ablation) | OpenCUA models · OSWorld, 15 steps | success rate | Helped |
| use L2 textual history rather than concise L1 history (ablation) | OpenCUA model with 3 visual history images · OSWorld | success rate | No clear effect |
| increase cross-domain training data from 7K Ubuntu to 10K Ubuntu plus 17K Win&Mac (ablation) | Qwen2-VL · OSWorld | success rate: 9.8\% to 18.5\% | Helped |
| increase Ubuntu training data from 3K to 10K trajectories (ablation) | Qwen2-VL · multiple benchmarks | average performance: 72\% | Helped |
| increase Win/Mac training data from 3K to 14K trajectories (ablation) | Qwen2-VL · multiple benchmarks | average performance: 125\% | Helped |
| increase test-time sampling from Pass@1 to Pass@16 | OpenCUA model · OSWorld, 15-step budget | success rate: 16.9 to 34.6 (+104\%) | Helped |
| increase test-time sampling from Pass@1 to Pass@16 | OpenCUA model · OSWorld, 50-step budget | success rate: 18.4 to 39.2 (+113\%) | Helped |
Surprising findings
- L2 (thought + action) reasoning at inference outperforms L1 action-only format, contrary to prior work (Aguvis) that found L1 better.
- Out-of-domain Win&Mac data improved OSWorld (Ubuntu) performance, from 9.8% to 18.5%, indicating cross-platform transfer rather than negative transfer.
Limitations
- Dataset scalability is bounded by human annotation effort and cost.
- Consent requirements introduce selection bias in the collected data.
- Online benchmark variance is high and models remain non-robust to small environment perturbations.
Experimental setup
Datasets: AgentNet ShowUI UGround Kimi general SFT data
Benchmarks: OSWorld-Verified WindowsAgentArena AgentNetBench OSWorld-G ScreenSpot-V2 ScreenSpot-Pro UI-Vision
Metrics: success rate step success rate pass@n
Baselines: OpenAI CUA Claude 4 Sonnet Claude Sonnet 4.5 UI-TARS-72B-DPO Qwen2.5-VL-72B Kimi-VL-A3B Aguvis-7B Seed1.5-VL
Builds on: Aguvis ActRe OSWorld Qwen2.5-VL Kimi-VL
Closest papers by meaning
- ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data (ICLR 2026 · similarity 0.92)
- Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.89)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.88)
- Computer Use at the Edge of the Statistical Precipice (NeurIPS 2026 · similarity 0.88)
- Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directedness (ICLR 2026 · similarity 0.88)
- GTA: A Benchmark for General Tool Agents (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.