Research map / Vision-language and multimodal / Robot action and navigation
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds ShowUI, a 2B vision-language-action model for GUI agents, using UI-guided visual token selection over an RGB connected-component graph, interleaved vision-language-action streaming, and a small balanced instruction-tuning dataset (256K samples) on Qwen2-VL-2B.
Problem
GUI visual agents face expensive high-resolution screenshot modeling, unclear action formulation across devices and interleaved vision-language-action history, and imbalanced, heterogeneous GUI training data.
Approach
Represent screenshot patches as nodes in an RGB-similarity connected-component graph (Union-Find) and randomly skip tokens within each component during training so self-attention runs on fewer tokens while keeping original position embeddings; combined with interleaved action-visual and action-query streaming, JSON action formatting, and balanced data sampling.
Key results
- ShowUI achieves 75.1% average zero-shot accuracy on Screenspot grounding with a 2B model.
- UI-guided token selection reduces visual tokens by about 33% during training and gives a 1.4x training speedup.
- On AITW mobile navigation ShowUI reaches 70.0 overall, above SeeClick (59.3) and Qwen2-VL-2B (67.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add UI-Graph-guided visual token selection (ablation) | ShowUI · Training efficiency | Training speedup: 1.4× | Helped |
| Add visual history to interleaved streaming (ablation) | ShowUI · AITW | Overall score: 68.3 -> 70.0 | Helped |
| Add visual history to interleaved streaming (ablation) | ShowUI · Mind2Web Cross-Task | Step SR: 36.9 -> 37.2 | No clear effect |
| Add visual history to interleaved streaming (ablation) | ShowUI · MiniWob 35-tasks split | Score: 70.4 -> 71.5 | Helped |
| Replace token merging with UI-Graph-guided token selection (ablation) | ShowUI visual token compression · Screenspot desktop subset | Accuracy: 34.7 -> 70.4 | Helped |
| Use UI-Graph routing instead of random token selection (ablation) | ShowUI visual token compression · Screenspot desktop subset | Accuracy: 65.3 -> 70.4 | Helped |
| Apply UI-Graph token selection at test time (ablation) | ShowUI visual token compression · Screenspot desktop subset | Accuracy: 70.4 -> 64.9 | Hurt |
| Insert token selection across alternating layers instead of early or late layers (ablation) | ShowUI visual token compression · Screenspot desktop subset | Accuracy: 68.2 -> 70.5; 67.6 -> 70.5 | Helped |
| Increase token selection ratio from 0.5 to 0.75 or 1.0 (ablation) | ShowUI visual token compression · Screenspot desktop subset | Accuracy: 70.4 -> 68.3; 70.4 -> 64.5 | Hurt |
| Train with grounding data only, excluding navigation data (ablation) | ShowUI · Screenspot | Avg. accuracy: 74.9 vs 75.1 | No clear effect |
| Add diverse query annotations to OmniAct (ablation) | ShowUI training data · Screenspot | Avg. accuracy: 68.7 -> 70.9 | Helped |
| Filter web training data to visual elements instead of text and visual elements (ablation) | ShowUI training data · Screenspot | Avg. accuracy: 66.6 -> 69.0 | Helped |
Surprising findings
- Token merging using the UI graph performed worse than random token selection (34.7 vs 56.2 on Screenspot), showing that losing positional information hurts grounding.
- Zero-shot ShowUI on MiniWob (27.1) trails fine-tuned Qwen-VL (48.4) by a wide margin despite strong zero-shot Mind2Web results.
Limitations
- Model is trained primarily on offline data and does not handle online environments well (MiniWob zero-shot 27.1).
- Test-time token selection reduces accuracy due to resolution loss.
- Desktop and web visual grounding data are scarce relative to mobile data.
Experimental setup
Datasets: Screenspot Mind2Web AITW MiniWob AMEX OmniAct GUIAct
Benchmarks: Screenspot Mind2Web AITW MiniWob
Metrics: accuracy Ele.Acc Op.F1 Step SR training speedup visual token count
Baselines: SeeClick CogAgent UGround OmniParser Qwen2-VL-2B Qwen-VL Pix2Act MindAct
Builds on: Qwen2-VL Mixture-of-Depth SeeClick Token Merging
Closest papers by meaning
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.92)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.92)
- Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining (ICML 2026 · similarity 0.90)
- GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.90)
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.90)
- VideoGUI: A Benchmark for GUI Automation from Instructional Videos (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.