Research map / Large language models / Agents, code and math
OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OSCAR, a state-machine GUI agent that controls desktop and mobile OSes by generating Python/PyAutoGUI code from A11Y-tree-grounded screenshots, with task-driven re-planning driven by execution and verification feedback.
Problem
Generalist agents struggle to operate across diverse desktop and mobile applications in dynamic, executable OS environments where real-time feedback and adaptive re-planning are needed.
Approach
Operate as a state machine (Init, Observe, Plan, Execute, Verify, Success, Fail, Reset, Error) that re-plans individual tasks rather than whole workflows, using dual GUI grounding (SoM visual marks from the accessibility tree plus semantic element labels) and code-based actions.
Key results
- On GAIA, OSCAR averages 28.7% exact match (Level 3: 13.5%) vs. MMAC's 24.0% average and 6.1% Level 3.
- On OSWorld with GPT-4o, OSCAR reaches 24.5% success rate vs. FRIDAY's 18.8% and UFO's 16.5%.
- On AndroidWorld, OSCAR achieves 61.6% average success vs. AppAgent's 59.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OSCAR instead of MMAC | GAIA · GAIA | exact match (Average): 24.0 -> 28.7 | Helped |
| Use OSCAR instead of MMAC | GAIA · GAIA Level 3 | exact match: 6.1 -> 13.5 | Helped |
| Use OSCAR instead of FRIDAY | OSWorld · OSWorld | success rate (Avg.): 18.8 -> 24.5 | Helped |
| Use OSCAR instead of AppAgent | AndroidWorld · AndroidWorld | success rate (Average): 59.9 -> 61.6 | Helped |
| Remove GUI-grounded dual input and use raw screenshots (ablation) | OSCAR · OSWorld | performance: 100% -> 70% | Hurt |
| Replace task-driven re-planning with direct prompts without fine-grained planning (ablation) | OSCAR · OSWorld | performance: 100% -> 80% | Hurt |
| Replace A11Y tree extraction with Detection+OCR (ablation) | OSCAR · OSWorld professional tools | performance | Hurt |
| Replace task-driven re-planning with ReAct (ablation) | OSCAR · OSWorld professional software and multi-application scenarios | performance | Hurt |
| Replace task-driven re-planning with Chain-of-Action (ablation) | OSCAR · OSWorld professional software and multi-application scenarios | performance | Hurt |
| Add HTML input | OSCAR · GUI-World Website | MC: 89.2 -> 92.2 | Helped |
Surprising findings
- ReAct and Chain-of-Action are comparable to OSCAR on daily-app and office scenarios, but fall behind on professional and multi-app tasks.
- Domain-specific GUI fine-tuning (CogAgent) performs poorly on multi-step navigation despite strong single-step understanding.
Limitations
- Relies on closed API models (GPT-4o/GPT-4-turbo) and a fixed maximum of 4 re-planning attempts.
- Depends on A11Y tree availability; Detection+OCR fallback is less effective on dense professional-tool screens.
Experimental setup
Datasets: GAIA OSWorld AndroidWorld GUI-World Mind2Web AITW
Benchmarks: GAIA OSWorld AndroidWorld GUI-World Mind2Web AITW
Metrics: exact match accuracy success rate step success rate action matching score BERTScore LLM-as-a-judge score multiple-choice accuracy
Baselines: Cradle UFO FRIDAY MMAC M3A AppAgent Mobile Agent GPT-4-turbo GPT-4 plugins ReAct Chain-of-Action Auto-GUI
Builds on: Set-of-Mark prompting plan-and-solve prompting ReAct chain-of-thought
Closest papers by meaning
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.86)
- Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolution (ICML 2026 · similarity 0.86)
- OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models (CVPR 2026 · similarity 0.86)
- Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.86)
- CoAct-1: Computer-using Multi-agent System with Coding Actions (ICLR 2026 · similarity 0.86)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.