Research map / Multi-agent systems and games / LLM agents
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UGround, a universal visual grounding model for GUI agents trained on 10M web-synthesized GUI elements with referring expressions, and SeeAct-V, a vision-only agent framework using it for pixel-level actions.
Problem
GUI agents rely on text-based representations (HTML, a11y trees) that are noisy, incomplete, and costly; vision-only agents need accurate, generalizable visual grounding that maps referring expressions to screen coordinates.
Approach
A simple recipe of web-based hybrid referring-expression synthesis (rules plus LLM/MLLM rewriting) and a LLaVA-NeXT architecture with dynamic high-resolution AnyRes input and direct numeric coordinate output, yielding a universal grounding model plugged into any planner in a two-stage vision-only agent (SeeAct-V).
Key results
- UGround reaches 73.3% average grounding accuracy on ScreenSpot (standard setting) vs 53.4% for SeeClick, and UGround-V1-72B reaches 89.4%.
- With GPT-4o planner on ScreenSpot agent setting, UGround achieves 81.4% average vs 52.3% for SeeClick.
- SeeAct-V with UGround on AndroidWorld reaches 32.8% success with GPT-4o, and UGround-V1-7B reaches 44.0%, exceeding text-based M3A variants (30.6%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add supplementary training data to Web-Hybrid (ablation) | UGround · ScreenSpot agent setting | average grounding accuracy: 76.9 -> 81.4 | Helped |
| train on supplementary data only instead of Web-Hybrid (ablation) | UGround · ScreenSpot agent setting | average grounding accuracy: 76.9 -> 73.4 | Hurt |
| replace the UGround model design with Qwen-VL while using Web-Hybrid (ablation) | UGround · ScreenSpot agent setting | average grounding accuracy: 76.9 -> 62.4 | Hurt |
| use dynamic image resolution instead of fixed square resolution (ablation) | UGround · ScreenSpot agent setting | average grounding accuracy: 72.2 -> 76.9 | Helped |
| increase fixed image resolution from 448 x 448 to 896 x 896 (ablation) | UGround · ScreenSpot agent setting | average grounding accuracy: 72.2 -> 73.8 | Helped |
| add positional referring expressions to Web-Hybrid (ablation) | UGround · ScreenSpot agent setting | average grounding accuracy: 76.0 -> 76.9 | Helped |
| add positional referring expressions to Web-Hybrid (ablation) | UGround · ScreenSpot standard | average grounding accuracy: 64.9 -> 68.8 | Helped |
| scale Web-Hybrid training data from 50K to 100K screenshots (ablation) | UGround · ScreenSpot agent setting | average grounding accuracy | Helped |
| scale Web-Hybrid data beyond 100K screenshots (ablation) | UGround · ScreenSpot agent setting | average grounding accuracy | Helped |
Surprising findings
- Web-only training transfers well to desktop UIs despite no desktop training data.
- Vision-only SeeAct-V with UGround matches or outperforms text-plus-SoM agents on several benchmarks.
Limitations
- Trained on very large synthetic data with redundancy, leaving data efficiency room for improvement.
- Long-tail idiosyncratic icons remain hard; no desktop UI training data limits desktop performance.
- UGround depends on an external planner and is not a standalone GUI agent.
Experimental setup
Datasets: Web-Hybrid Web-Direct GUIAct AndroidControl Widget Caption UIBert AITZ Common Crawl
Benchmarks: ScreenSpot Multimodal-Mind2Web AndroidControl OmniACT Mind2Web-Live AndroidWorld
Metrics: grounding accuracy element accuracy step accuracy action score completion rate task success rate
Baselines: SeeClick CogAgent GPT-4 GPT-4o DetACT M3A Choice grounding SoM
Builds on: LLaVA-NeXT SeeAct SeeClick Set-of-Mark
Closest papers by meaning
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.92)
- Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.92)
- GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents (NeurIPS 2025 · similarity 0.91)
- Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis (NeurIPS 2025 · similarity 0.91)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.91)
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.