Research map / Multi-agent systems and games / LLM agents
CoAct-1: Computer-using Multi-agent System with Coding Actions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoAct-1, a multi-agent computer-use system where an Orchestrator delegates subtasks to either a GUI Operator or a code-writing Programmer, achieving state-of-the-art on OSWorld (60.76%) and WindowsAgentArena (52.5%).
Problem
GUI-only computer-use agents are inefficient and brittle on long-horizon tasks such as file management and spreadsheet processing, where long sequences of clicks are error-prone.
Approach
Extend the agent action space with free-form Python/Bash coding actions alongside GUI actions, and use a high-level Orchestrator to dynamically route each subtask to a Programmer agent or a GUI Operator, with isolated memories and summarized handoffs.
Key results
- CoAct-1 reaches 60.76% success on OSWorld within a 150-step budget, above Agent S2.5 w/ o3 (55.98%) and GTA-1 w/ o3 (53.07%).
- CoAct-1 achieves 52.5% on WindowsAgentArena at 100 steps versus 29.8% for Agent S2.
- Average steps per task on OSWorld is 10.15 versus about 15 for GTA-1; a Programmer-only ablation averages 1.14 steps but reaches only 35.73% success.
Surprising findings
- Programmer-only agent is very step-efficient (1.14 steps per success) but reaches only 35.73% success, showing many tasks still need GUI interaction.
- Swapping the Orchestrator and Programmer from o4-mini to o3 raises OSWorld success from 43.43% to 58.72%, indicating planner strength dominates.
Limitations
- Relies on closed OpenAI models (o3, o4-mini, computer-use-preview); open-source substitutes not tested due to lack of GPU resources.
- Only the final OS state is returned to the Orchestrator, so intermediate GUI errors can be missed.
- Failures from high-level or ambiguous queries and hallucinations by the Orchestrator and GUI Operator.
Experimental setup
Datasets: OSWorld WindowsAgentArena
Benchmarks: OSWorld WindowsAgentArena
Metrics: success rate average steps
Baselines: OpenAI o3 UI-TARS-1.5 (7B) OpenAI CUA 4o OpenCUA Jedi-7B w/ o3 GTA-1 w/ o3 Agent S2.5 w/ o3 Agent S NAVI Agent S2
Builds on: Agent S2.5 AG2 GTA-1 OSWorld
Closest papers by meaning
- Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.88)
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.88)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.87)
- AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · similarity 0.87)
- OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.87)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.