Research map / Multi-agent systems and games / LLM agents
Agent S: An Open Agentic Framework that Uses Computers Like a Human
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Agent S, a GUI agent framework combining experience-augmented hierarchical planning (web knowledge plus narrative/episodic memory), a self-evaluator, and an Agent-Computer Interface, evaluated on OSWorld and WindowsAgentArena.
Problem
Autonomous GUI agents struggle with domain-specific knowledge, long-horizon multi-step planning, and dynamic non-uniform desktop interfaces when controlling computers via keyboard and mouse.
Approach
Hierarchical Manager/Worker planning that fuses online web search and narrative memory for plans, uses episodic memory of subtask experiences for execution, updates memories continually via a self-evaluator that stores textual summaries, and grounds actions through a language-centric ACI with bounded primitive actions and an OCR-augmented accessibility tree.
Key results
- On the full OSWorld test set (369 tasks), Agent S with GPT-4o reaches 20.58% overall success versus 11.21% for the GPT-4o baseline (83.6% relative improvement).
- Agent S with Claude-3.5-Sonnet reaches 20.48% overall on OSWorld.
- On WindowsAgentArena (154 tasks, GPT-4o), Agent S achieves 18.2% overall versus 13.3% for NAVI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Agent S instead of the GPT-4o baseline | GPT-4o GUI agent · OSWorld full test set | success rate: 11.21 -> 20.58 | Helped |
| use Agent S instead of the GPT-4o baseline | GPT-4o GUI agent · WindowsAgentArena full test set | success rate: 13.3 -> 18.2 | Helped |
| add web knowledge (ablation) | Agent S · OSWorld test_sub | success rate: 16.80 -> 26.15 | Helped |
| add narrative memory (ablation) | Agent S · OSWorld test_sub | success rate: 21.41 -> 26.15 | Helped |
| add episodic memory (ablation) | Agent S · OSWorld test_sub | success rate: 18.46 -> 26.15 | Helped |
| add web knowledge, narrative memory, and episodic memory (ablation) | Agent S without experience components · OSWorld test_sub | success rate: 13.85 -> 26.15 | Helped |
| add retrieval (ablation) | baseline GUI agent · OSWorld test_sub | success rate: 10.77 -> 12.31 | Helped |
| add retrieval (ablation) | Agent S (ACI-only) · OSWorld test_sub | success rate: 12.31 -> 20.00 | Helped |
| add hierarchical planning (ablation) | ACI-only + Experiential Learning · OSWorld test_sub | success rate: 20.00 -> 26.15 | Helped |
| add continual memory updates (ablation) | Agent S using exploration memory · OSWorld test_sub | success rate: 23.08 -> 26.15 | Helped |
| add the Self-Evaluator to summarize experiences (ablation) | Agent S · OSWorld test_sub | success rate: 18.46 -> 26.15 | Helped |
| add self-supervised exploration (ablation) | Agent S · OSWorld test_sub | success rate: 15.38 -> 26.15 | Helped |
Surprising findings
- Removing web knowledge caused the largest ablation drop (26.15% to 16.80% on the 65-task subset), and self-supervised exploration mattered more than continual memory update.
- Execution errors were the most common failure type (79.59% of failed tasks), with grounding and planning errors contributing to many of them.
Limitations
- Relies entirely on closed proprietary models (GPT-4o, Claude-3.5-Sonnet); no open-weight evaluation.
- Ablations run on a 65-task subset due to inference cost, and step count / wall-clock time are not measured.
- Execution and grounding errors remain common, causing repetitive actions.
Experimental setup
Datasets: OSWorld WindowsAgentArena
Benchmarks: OSWorld WindowsAgentArena
Metrics: success rate
Baselines: OSWorld agent (GPT-4o) Claude-3 Gemini-Pro-1.5 GPT-4V NAVI
Builds on: OSWorld Chain-of-thought prompting Set-of-Mark prompting
Closest papers by meaning
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.91)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.91)
- AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · similarity 0.90)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.90)
- DocOS: Towards Proactive Document-Guided Actions in GUI Agents (ICML 2026 · similarity 0.90)
- AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.