Research map / Vision-language and multimodal / Multimodal LLMs

Learning GUI Grounding with Spatial Reasoning from Visual Feedback

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reframes GUI grounding as an interactive multi-step cursor-moving search, trained with GRPO and a dense trajectory-penalized reward, reaching 56.5-58.1% on ScreenSpot-Pro with 8K training samples.

Problem

VLMs often fail to predict accurate numeric screen coordinates for GUI grounding on high-resolution, complex-layout screenshots, and one-step coordinate prediction training lacks a visual feedback loop linking coordinates to on-screen positions.

Approach

Treat GUI grounding as an interactive search: the VLM sees a rendered cursor, reasons about its spatial relation to the target, and iteratively moves it, trained with multi-step online RL (GRPO) using a position reward plus four trajectory penalties (false stop, false move, false direction, repeated position); at inference a cursor-centric focusing crop is applied.

Key results

• GUI-Cursor-7B (UI-TARS-1.5-7B) reaches 58.1% average on ScreenSpot-Pro, outperforming GTA1-7B (50.1%) using 8K vs 64K training samples.
• GUI-Cursor-7B (Qwen2.5-VL-7B) reaches 93.9% on ScreenSpot-v2 and 56.5% on ScreenSpot-Pro; on OSWorld with o3 planner it gets 57.1% at 50 steps vs GTA1-7B 53.1% at 100 steps.
• Zero-shot Qwen2.5-VL-7B drops from 88.8% one-step ScreenSpot-v2 accuracy to 36.3% with 4-step direct cursor moves, showing single-step accuracy does not transfer to iterative moving.

Surprising findings

• Qwen2.5-VL-7B shows a strong centre bias on a cursor-in-box test and fails near image edges, suggesting its grounding is not founded on robust spatial understanding.
• Relative cursor movement with Qwen2.5-VL-7B yields near-zero accuracy, though GPT-4o benefits from it.

Limitations

• Cursor-centric focusing fails when the initial coarse prediction is far from the target (about 10.3% of ScreenSpot-Pro cases), since no zoom-out mechanism exists.
• Multi-step RL rollouts are time-consuming, and training used max 4 steps only because of cost.
• Gains over one-step baselines are modest on easier benchmarks such as ScreenSpot-v2.

Experimental setup

Datasets: Aria-UI OS-Atlas ScreenSpot-V2 ScreenSpot-Pro OSWorld-G UI-Vision SPHERE SpatialMQA OSWorld

Benchmarks: ScreenSpot-Pro ScreenSpot-v2 OSWorld-G UI-Vision OSWorld SPHERE SpatialMQA cursor-in-box test

Metrics: accuracy F1 success rate

Baselines: SeeClick OS-Atlas-7B UGround-7B UI-TARS-7B UI-TARS-72B Jedi-7B GUI-Actor-7B UI-R1-3B UI-R1-E-3B GUI-R1-3B InfiGUI-R1-3B GUI-R1-7B

Builds on: GRPO SE-GUI dense position reward GTA1 EasyR1

Closest papers by meaning

• GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.92)
• GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.91)
• GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding (ICML 2026 · similarity 0.91)
• GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning (ICLR 2026 · similarity 0.90)
• GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents (NeurIPS 2025 · similarity 0.90)
• Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.90)