Research map / Vision-language and multimodal / Human and embodied modeling
Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a GUI grounding benchmark (with fine-grained capability categories and refusal cases) and a large synthetic grounding dataset built by decomposing UI knowledge into icon, component, and layout data, then fine-tunes Qwen2.5-VL models that improve ScreenSpot-v2/Pro and agent success on OSWorld.
Problem
GUI grounding benchmarks reduce grounding to short referring expressions and lack data covering icons, layouts, fine-grained manipulation, and infeasible instructions, limiting computer-use agents.
Approach
Decompose GUI grounding into icon, component, and layout data sources; synthesize large-scale grounding data from code rendering, design templates, OS rollouts and icon libraries; and add refusal examples built by mismatching instructions with unrelated screenshots, then fine-tune Qwen2.5-VL.
Key results
- Qwen2.5-VL-7B fine-tuned on the dataset reaches 91.7% average on ScreenSpot-v2 and 39.5% on ScreenSpot-Pro (per tables reported).
- With GPT-4o as planner and the 7B grounder, OSWorld success rises to 27.0%; with o3 as planner it reaches 51.0%, versus 23% for o3 alone in the paper's baseline table.
- On the new benchmark, the 7B model reaches 54.1% overall, versus 31.4% for Qwen2.5-VL-7B and 47.5% for UI-TARS-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with increasing proportions of the mixed icon, component, and layout data (ablation) | Qwen2.5-VL-3B · ScreenSpot-v2 and benchmark | performance | Helped |
| Scale up a single data type rather than mixed data types (ablation) | Qwen2.5-VL-3B · ScreenSpot-v2 and benchmark | performance stability | Hurt |
| Combine specialized data types rather than scale a single data type (ablation) | Qwen2.5-VL-3B · ScreenSpot-v2 and benchmark | performance stability | Helped |
| Refine instructions with precise, visually identifiable descriptions (ablation) | GUI grounding models · OSWorld Grounding Benchmark | accuracy | Helped |
| Use Scaling Computer-Use Grounding 7B rather than Aguvis-72B as the grounder | GPT-4o agent on OSWorld · OSWorld | success rate: 17.0 -> 27.0 | Helped |
Surprising findings
- Despite refusal training data, the models rarely produce refusal responses; only Gemini-2.5-Pro among evaluated models refused often.
- After instruction refinement, the 3B/7B model reaches performance comparable to UI-TARS-72B, suggesting grounding is limited more by knowledge than model size.
Limitations
- Refusal ability remains weak; models rarely output refusals despite refusal training.
- Data is limited to synthetic and collected sources; scaling via internet screenshots/videos is left for future work.
- Benchmark uses a bounding-box containment criterion and a relatively small number of annotated examples.
Experimental setup
Datasets: ScreenSpot-v2 ScreenSpot-Pro UI-Vision Aguvis SeeClick OS-Atlas OmniParser-v2 layout rollouts Figma community templates Material UI / Ant Design / Chakra component code
Benchmarks: ScreenSpot-v2 ScreenSpot-Pro UI-Vision OSWorld WindowsAgentArena new grounding benchmark (OSWorld-Grounding)
Metrics: accuracy (point in bounding box) success rate
Baselines: SeeClick OS-Atlas-7B UI-TARS-7B UI-TARS-72B Operator Qwen2.5-VL-3B Qwen2.5-VL-7B Qwen2.5-VL-32B OmniParser-v2 UGround-V1-7B Aguvis-7B Gemini-2.5-Pro
Builds on: Qwen2.5-VL Aguvis OS-Atlas SeeClick OmniParser
Closest papers by meaning
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.91)
- UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction as Reasoning (ICLR 2026 · similarity 0.91)
- Grounding Computer Use Agents on Human Demonstrations (ICLR 2026 · similarity 0.90)
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.90)
- WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments (ICML 2026 · similarity 0.89)
- Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.