Research map / Vision-language and multimodal / Multimodal LLMs
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GUI-Actor, a coordinate-free GUI grounding VLM that uses an attention-based action head over a special <ACTOR> token and a separate grounding verifier to rerank candidate regions, reaching SOTA on ScreenSpot-family benchmarks.
Problem
Visual grounding for VLM-powered GUI agents is usually framed as text-based coordinate generation, which suffers from weak spatial-semantic alignment, ambiguous single-point supervision, and a mismatch between dense screen coordinates and patch-level visual features.
Approach
Replace coordinate tokens with a dedicated <ACTOR> token whose final hidden state attends over visual patch tokens via an action head trained with multi-patch bounding-box supervision (KL to a uniform distribution over patches overlapping the target box); a fine-tuned True/False verifier then selects among top-K attention candidates.
Key results
- GUI-Actor-7B with Qwen2-VL reaches 40.7 on ScreenSpot-Pro (44.2 with verifier), above UI-TARS-72B at 38.1; with Qwen2.5-VL backbone it reaches 44.6 (47.7 with verifier).
- GUI-Actor-2B reaches 86.5 on ScreenSpot and 88.6 on ScreenSpot-v2, and GUI-Actor-7B reaches 89.5 on ScreenSpot-v2 (89.7 and 88.3 on ScreenSpot respectively for 7B, 2B+verifier 89.3 on v2).
- With a frozen backbone (LiteTrain, ~100M trainable params for 7B) plus verifier, ScreenSpot-Pro reaches 35.8 and ScreenSpot-v2 83.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the grounding verifier | GUI-Actor-7B · ScreenSpot-Pro | Element Accuracy (Avg): 40.7 -> 44.2 | Helped |
| Add the grounding verifier | GUI-Actor-2B · ScreenSpot | Element Accuracy (Avg): 86.5 -> 86.9 | Helped |
| Add the grounding verifier | GUI-Actor-7B · ScreenSpot-v2 | Element Accuracy (Avg): 89.5 -> 90.9 | Helped |
| Freeze the backbone and train only the action head and special tokens (ablation) | GUI-Actor-7B · ScreenSpot-Pro | Element Accuracy (Avg): 40.7 -> 22.9 | Hurt |
| Add the verifier to frozen-backbone LiteTrain (ablation) | GUI-Actor-7B-LiteTrain · ScreenSpot-Pro | Element Accuracy (Avg): 22.9 -> 35.8 | Helped |
| Replace coordinate-free action grounding with bounding-box coordinate generation (ablation) | GUI-Actor-7B · ScreenSpot-Pro | Element Accuracy (Avg): 40.7 -> 13.8 | Hurt |
| Replace coordinate-free action grounding with point coordinate generation (ablation) | GUI-Actor-7B · ScreenSpot-Pro | Element Accuracy (Avg): 40.7 -> 15.6 | Hurt |
| Use verifier self-aggregation across image crops (ablation) | GUI-Actor-7B + Verifier · ScreenSpot-Pro | Element Accuracy (Avg): 44.2 -> 44.5 | Helped |
| Add the verifier | GUI-Actor-7B with Qwen2.5-VL backbone · ScreenSpot-Pro | Element Accuracy (Avg): 44.6 -> 47.7 | Helped |
Surprising findings
- Aguvis-7B trained with bounding-box supervision performs similar to or worse than point supervision, suggesting coordinate generation does not benefit from box annotations without architectural spatial bias.
- Freezing the entire VLM and training only the action head plus verifier achieves performance close to fully fine-tuned coordinate-based models.
Limitations
- Very small interface elements (under ~10x10 px) are poorly represented given the 28x28 patch size of Qwen2-VL.
- UI-TARS-7B outperforms GUI-Actor-7B on ScreenSpot and ScreenSpot-v2 and is trained on a larger proprietary-inclusive dataset with DPO.
- Verifier-based selection adds a second model call per candidate, and the verifier is only lightly evaluated against other verifiers.
Experimental setup
Datasets: UGround Web-Hybrid GUI-Env GUI-Act AndroidControl AMEX Wave-UI OS-Atlas SeeClick
Benchmarks: ScreenSpot-Pro ScreenSpot ScreenSpot-v2 OSWorld-W
Metrics: element accuracy task success rate
Baselines: Aguvis-7B UI-TARS-7B UI-TARS-2B UGround-V1-7B OS-Atlas-7B Jedi-7B GPT-4o NAVI OmniAgent
Builds on: Aguvis Qwen2-VL UI-TARS
Closest papers by meaning
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.91)
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.91)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.90)
- Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.90)
- Learning GUI Grounding with Spatial Reasoning from Visual Feedback (ICML 2026 · similarity 0.90)
- GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.