Research map / Multi-agent systems and games / LLM agents
Cradle: Empowering Foundation Agents towards General Computer Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cradle, a modular LMM-powered framework (six modules, GPT-4o backbone) that lets foundation agents control arbitrary software and games from screenshots via keyboard and mouse, evaluated on four commercial games, five apps, and OSWorld.
Problem
Foundation agents generalize poorly across virtual environments because each environment has hand-designed observation and action spaces and built-in APIs; the paper targets general computer control using only screenshots and keyboard/mouse actions.
Approach
A modular LMM-driven loop (information gathering, self-reflection, task inference, skill curation, action planning, memory) that generates executable code skills to map high-level plans to low-level keyboard/mouse operations, with no built-in APIs, under a unified General Computer Control setting.
Key results
- On OSWorld, Cradle reaches 7.81% overall success vs 5.03% for GPT-4o and 4.59% for GPT-4o+SoM, and 20.41% on the professional domain.
- In Cities: Skylines, the agent builds a city with a maximal population of 450±224, rising to 850±142 with human assistance on water pipes.
- In Dealer's Life 2, the agent achieves a total profit rate of +39.6% averaged over runs, peaking at +87.4% in one run.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Cradle instead of GPT-4o alone | OSWorld · OSWorld All | success rate (%): 5.03 -> 7.81 | Helped |
| Add SoM to GPT-4o | OSWorld · OSWorld All | success rate (%): 5.03 -> 4.59 | Hurt |
| Add self-reflection and episodic memory (ablation) | React-like · RDR2 Follow Micah | successful runs: (1/5) -> (2/5) | Helped |
| Add episodic summary and task inference (ablation) | Cradle · Stardew Valley Cultivation | successful harvest | Helped |
| Replace visual input with a textual description (ablation) | Voyager-like · RDR2 Follow Dutch | successful runs: (5/5) -> (3/5) | Hurt |
| Replace GPT-4o with Claude 3 Opus (ablation) | Cradle · RDR2 Follow Micah | successful runs: (5/5) -> (4/5) | Hurt |
| Use Cradle instead of React-like | RDR2 Follow Dutch · RDR2 Follow Dutch | interaction steps (successful runs): 15 ± 2 -> 13 ± 3 | Helped |
| Use Cradle instead of Reflextion-like | RDR2 Follow Micah · RDR2 Follow Micah | successful runs: (2/5) -> (5/5) | Helped |
| Add information gathering and self-reflection modules | Cradle · OSWorld Professional | success rate (%): 4.08 -> 20.41 | Helped |
Surprising findings
- GPT-4o struggles to recognize and locate objects in the 2D pixel-art Stardew Valley, leading to poor shopping performance (1/5).
- Claude 3 Opus underperforms GPT-4o as backbone, attributed to unreliable OCR for guidance text.
Limitations
- Struggles with out-of-distribution icons and non-realistic game styles such as Stardew Valley.
- Frequent LMM calls create high cost and latency; no audio modality.
- No training or self-improvement; relies entirely on prompting GPT-4o, and evaluation is mostly human judgment.
Experimental setup
Datasets: OSWorld
Benchmarks: OSWorld Red Dead Redemption 2 missions Stardew Valley tasks Dealer's Life 2 weekly shop Cities: Skylines city building Chrome/Outlook/CapCut/Meitu/Feishu tasks
Metrics: success rate interaction steps profit rate population
Baselines: GPT-4o GPT-4o+SoM ReAct-like Reflexion-like Voyager-like Claude 3 Opus
Builds on: Voyager ReAct Reflexion SIMA
Closest papers by meaning
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.85)
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.84)
- AgentStudio: A Toolkit for Building General Virtual Agents (ICLR 2025 · similarity 0.84)
- OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning (ICLR 2025 · similarity 0.83)
- CoAct-1: Computer-using Multi-agent System with Coding Actions (ICLR 2026 · similarity 0.83)
- GTA1: GUI Test-time Scaling Agent (ICLR 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.