Research map / Vision-language and multimodal / Robot action and navigation

UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction

ICML 2025 · Vision-language and multimodal › Robot action and navigation · approach: Embodied multimodal LLM agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces UI-Vision, a licensed desktop-centric GUI benchmark evaluating element grounding (basic, functional, spatial) and layout grounding/action prediction for GUI agents, reporting that even strong models score low.

Problem

Desktop GUI agents are underexplored due to data collection difficulty and licensing issues, and existing benchmarks lack evaluation of visual perception and grounding on desktop interfaces.

Approach

A comprehensive, licensed desktop GUI benchmark with fine-grained element grounding tasks (basic, functional, spatial categories) plus layout grounding and action prediction, evaluated across closed VLMs, open VLMs, and GUI agents.

Key results

• UI-TARS-72B achieves the best overall element grounding score of 25.5% final average, with 31.4% basic, 30.5% functional, and 14.7% spatial.
• GPT-4o scores only 1.38% final average on element grounding, while Claude-3.7-Sonnet reaches 8.27%.
• Spatial grounding is the hardest setting; the best open GUI agent UI-TARS-7B reaches 8.37% spatial overall versus 17.6% final average.

Surprising findings

• General closed-source VLMs such as GPT-4o score near zero on element grounding, far below specialized GUI agents.
• Spatial reasoning for grounding is far harder than basic or functional grounding across all models.

Limitations

• Results shown are only for element grounding in this excerpt; other task results are not included in the provided text.
• Closed-source models evaluated via API, so results depend on provider versions.

Experimental setup

Datasets: UI-Vision

Benchmarks: UI-Vision

Metrics: accuracy (%)

Baselines: GPT-4o Claude-3.5-Sonnet Claude-3.7-Sonnet Gemini-1.5-pro Gemini-Flash-2.0 Qwen-2.5VL-7B InternVL2.5-8B Qwen-2VL-7B MiniCPM-V-8B ShowUI-2B AriaUI-25.3B UGround-v1-7B

Closest papers by meaning

• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.92)
• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.91)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.91)
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction (ICML 2025 · similarity 0.90)
• GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.89)
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.89)