Research map / Multi-agent systems and games / LLM agents

WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments

ICML 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces WinDeskGround, a parametric multi-window desktop synthesis framework and 1,356-pair benchmark for evaluating GUI grounding robustness under occlusion, layout density, and semantic distractors, finding top agents degrade sharply under occlusion.

Problem

MLLM-based GUI grounding is mostly evaluated on idealized single-window screenshots, leaving robustness in cluttered, multi-window desktop environments (stacking, occlusion, visually similar distractors) under-measured.

Approach

Synthesizes controllable multi-window desktops from high-resolution single-window screenshots by parametrically varying window count, target visibility (occlusion), and semantic similarity of distractor windows, enabling factor-level robustness analysis instead of natural-distribution evaluation.

Key results

• Single Window accuracy: UGround, UI-TARS, and InfiGUI exceed 80%, while SeeClick starts near 25%.
• At L4, InfiGUI reaches 21.39% and is the most robust model; at L5 UI-TARS and InfiGUI retain ~10-11% while UGround drops to 7.08%.
• Under occlusion, UGround, UI-TARS, and InfiGUI fall below 20% when target visibility is 30-50%, from over 75% at full visibility.

Surprising findings

• Semantic similarity of distractors produced nearly flat accuracy curves, since distractors were placed behind the target.
• Increasing window count (visual clutter) did not degrade top-tier models much, contrary to expectation.

Limitations

• Static single-step snapshots; no temporal dynamics, hover, or focus changes.
• Synthetic composition may introduce layering and spatial-prior biases.
• Human approval of instruction validity is only 85% under strict criteria, and L4-L5 quality is lower.

Experimental setup

Datasets: Windows desktop screenshots collected by authors (585 screenshots)

Benchmarks: WinDeskGround

Metrics: Click Accuracy Euclidean distance Hit Rate

Baselines: SeeClick OS-Atlas-7B UGround-7B UI-TARS-1.5-7B InfiGUI-G1-7B

Builds on: Set-of-Mark ScreenSpot-Pro OmniParser

Closest papers by meaning

• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.90)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis (NeurIPS 2025 · similarity 0.89)
• OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.89)
• Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.88)
• GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding (ICML 2026 · similarity 0.88)
• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.88)