Research map / Multi-agent systems and games / LLM agents
Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Aguvis is a unified pure-vision GUI agent trained in two stages (grounding, then planning and reasoning with inner monologue) on a cross-platform dataset, built on Qwen2-VL and achieving state-of-the-art on ScreenSpot, Multimodal-Mind2Web, AndroidControl and online benchmarks without closed-source models.
Problem
GUI agents depend on lengthy textual representations (HTML, accessibility trees), heterogeneous platform-specific action spaces, weak visual grounding, reliance on closed-source LLMs for reasoning, and reactive low-level action generation.
Approach
Operate purely on screenshots with a unified pyautogui-based action space plus pluggable platform actions, and train a two-stage pipeline (grounding, then planning with GPT-4o-generated inner monologue) so that an open-source VLM acts as a fully autonomous GUI agent.
Key results
- Aguvis-72B reaches 89.2 average on ScreenSpot, the best reported in the paper.
- On OSWorld, GPT-4o planner with Aguvis-72B grounder reaches 17.04% task success, versus 4.59% for GPT-4o with SoM and 14.9% for Claude Computer-Use.
- On Mind2Web-Live, Aguvis-72B achieves 27.1% SR at cost 0.012 per successful step, versus 22.1% for GPT-4o with Choice grounding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Stage 2 planning and reasoning training (ablation) | Aguvis-7B · Multimodal Mind2Web, Cross-Task | Step SR: 50.9 -> 60.4 | Helped |
| add Stage 2 planning and reasoning training (ablation) | Aguvis-7B · AndroidControl, low-level | Step Acc.: 75.6 -> 80.5 | Helped |
| add Stage 1 grounding training (ablation) | Qwen2-VL · ScreenSpot | accuracy: 77.4 -> 84.4 | Helped |
| add Stage 2 training after Stage 1 (ablation) | Aguvis-7B · ScreenSpot | accuracy: 81.8 -> 84.4 | Helped |
| add inner monologue to training (ablation) | Aguvis-7B · ScreenSpot | accuracy: 79.3 -> 84.4 | Helped |
| add inner monologue to training (ablation) | Aguvis-7B · AndroidControl, low-level | Step Acc.: 69.1 -> 80.5 | Helped |
| train with web and mobile data instead of web-only data (ablation) | Aguvis · Multimodal Mind2Web | Task: 53.1 -> 58.5 | Helped |
| use grounding packing strategy (ablation) | Aguvis grounding training · ScreenSpot website split | accuracy: 73.3 to 76.8 | Helped |
| use grounding packing strategy (ablation) | Aguvis grounding training · website grounding training | GPU hours: 6 hours to 1 hour | Helped |
| increase maximum image resolution from 1280×720 to 1920×1080 (ablation) | Aguvis Qwen2-VL · ScreenSpot | accuracy | No clear effect |
| apply the full Aguvis training process (ablation) | LLaVA · ScreenSpot | accuracy: 3.8% to 81.2% | Helped |
| prompt the agent to generate inner monologue before execution | Aguvis · ScreenSpot grounding-error cases | grounding errors resolved: 20% | Helped |
Surprising findings
- Training only on web and mobile trajectories transfers to desktop OSWorld, reaching 17.04% task success with a GPT-4o planner.
- Removing the inner monologue reduces ScreenSpot performance from 84.4 to 79.3 and low-level AndroidControl from 80.5 to 69.1.
Limitations
- Relies on GPT-4o to generate inner monologue annotations, and 13.3% of sampled augmented data was judged not aligned.
- Model cannot express uncertainty or refuse ambiguous instructions; 40% of ScreenSpot errors stem from ambiguous instructions.
- Mind2Web-Live evaluation was affected by automation detection, reCAPTCHA and network errors that prevented scoring 24 tasks.
Experimental setup
Datasets: SeeClick data GUIEnv GUIAct WebUI Widget Captioning RicoSCA UI RefExp RICO Icon OmniACT MM-Mind2Web MiniWoB++ AitZ
Benchmarks: ScreenSpot Multimodal-Mind2Web AndroidControl Mind2Web-Live AndroidWorld MobileMiniWob OSWorld
Metrics: grounding accuracy element accuracy operation F1 step success rate step accuracy task success rate cost per successful step
Baselines: GPT-4o GPT-4 CogAgent SeeClick UGround OmniParser Claude Computer-Use OpenAI Operator
Builds on: Qwen2-VL LLaVA-OneVision SeeClick Multimodal-Mind2Web AndroidWorld
Closest papers by meaning
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.90)
- GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.90)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.89)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.89)
- Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.88)
- SpiritSight Agent: Advanced GUI Agent with One Look (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.