Research map / Multi-agent systems and games / LLM agents

ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds ScaleCUA, a cross-platform computer-use dataset from a hybrid agent-human pipeline and trains open 3B/7B/32B Qwen2.5-VL-based computer use agents that set new results on GUI understanding, grounding, and web task completion.

Problem

Computer use agents are limited by the scarcity of large-scale, open, cross-platform GUI data (especially action trajectories) and by the limited GUI transferability of existing VLMs.

Approach

A dual-loop data pipeline (agent-environment interaction with rule-based DFS random-walk agents, plus agent-human hybrid trajectory collection) unified under a cross-platform action space, used to SFT Qwen2.5-VL into ScaleCUA models supporting grounding, direct-action, and reasoned-action modes.

Key results

• ScaleCUA-32B reaches 94.4% on MMBench-GUI L1-Hard and 60.6% on OSWorld-G, and 47.4% success on WebArena-Lite-v2 at 50 steps (+26.0 over UI-TARS-72B-DPO).
• ScaleCUA-32B achieves 94.7% average on ScreenSpot-v2 and 59.2% on ScreenSpot-Pro.
• ScaleCUA-3B reaches 89.2% on ScreenSpot-v2, outperforming Qwen2.5-VL-7B by 0.4 points.

Surprising findings

• Adding general multimodal data improves general VLM benchmarks but degrades GUI grounding accuracy, indicating conflicting optimization signals.
• Higher training resolution (2K) improves ScreenSpot-Pro and OSWorld-G grounding but slightly reduces online agent success rates on OSWorld and AndroidWorld.

Limitations

• Rule-based exploration yields semantically weak trajectories; agent-collected data quality lags expert data.
• No reflection, memory, hierarchical planning, PRM or RL optimization explored; memory is rudimentary flat history.
• Models fail on procedural prerequisites and silent action failures, with weak error recovery; macOS results remain low.

Experimental setup

Datasets: ScaleCUA-Data SeeClick UGround OS-Atlas JEDI AndroidControl GUI Odyssey AitZ Mind2Web AgentNet InternVL3 post-training data

Benchmarks: MMBench-GUI L1 MMBench-GUI L2 ScreenSpot-v2 ScreenSpot-Pro OSWorld-G AndroidControl AndroidWorld OSWorld WindowsAgentArena MacOSArena WebArena-Lite-v2 ScienceBoard

Metrics: accuracy success rate

Baselines: UI-TARS-72B-DPO Qwen2.5-VL-72B GUI-Owl-32B GUI-Owl-7B JEDI-7B OpenCUA-32B GPT-4o UGround-V1-7B

Builds on: Qwen2.5-VL OmniParser AGUVIS OS-Genesis

Closest papers by meaning

• OpenCUA: Open Foundations for Computer-Use Agents (NeurIPS 2025 · similarity 0.92)
• TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution (ICML 2026 · similarity 0.87)
• Computer Use at the Edge of the Statistical Precipice (NeurIPS 2026 · similarity 0.86)
• OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.86)
• SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? (NeurIPS 2026 · similarity 0.85)
• GTA1: GUI Test-time Scaling Agent (ICLR 2026 · similarity 0.85)