Research map / Vision-language and multimodal / Multimodal LLMs

GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a 7B GUI grounding model (GUI-Spotlight) to iteratively crop, quadrant-extract, and color-find on screenshots via SFT warm-up plus a modified GSPO RL, reaching 52.8% on ScreenSpot-Pro with 18.5K samples.

Problem

Visual grounding of textual references to exact on-screen elements in high-resolution GUI screenshots remains unreliable for 7B GUI agents, limiting precise pointer actions.

Approach

A think-with-image grounding model that dynamically invokes crop, extract and find_color tools to iteratively narrow focus, trained with SFT warm-up and a GSPO variant augmented by an auxiliary cross-entropy term on format-valid, correct trajectories (plus bucketed tool-balanced sampling in stage 3).

Key results

• GUI-Spotlight (init. UI-TARS-1.5-7B) reaches 52.8% on ScreenSpot-Pro with 18.5K samples, versus V2P-7B 50.6% (9.6M) and GTA-1-7B 50.1% (1.56M).
• On UI-Vision, the UI-TARS-initialized variant reaches 23.4% average, +5.3 over UI-TARS-1.5-7B (18.1%).
• On OSWorld-G, the UI-TARS-initialized variant averages 62.7%, versus UI-TARS-1.5-7B at 61.9%.

Surprising findings

• Vanilla GRPO or GSPO collapses around 300 RL steps as tool-call syntax degrades; adding an auxiliary cross-entropy term on correct, format-valid trajectories prevents this.
• Selecting only the most uncertain prompts and continuously updating the reference policy both degraded accuracy.

Limitations

• Qwen2.5-VL-7B-initialized variant is far weaker than UI-TARS-initialized one (38.7 vs 52.8 on ScreenSpot-Pro; 8.3 vs 23.4 on UI-Vision), so gains depend on UI-specific initialization.
• Model initially has virtually no multi-step reasoning capacity, requiring SFT warm-up on tool trajectories generated by a 72B model.

Experimental setup

Datasets: UGround ScreenSpot-Pro UI-Vision OSWorld-G

Benchmarks: ScreenSpot-Pro UI-Vision OSWorld-G

Metrics: accuracy

Baselines: UI-TARS-1.5-7B GTA-1-7B V2P-7B UI-Venus-7B GUI-Actor-2.5VL-7B SE-GUI-7B Qwen2.5-VL-7B-Instruct UI-TARS-72B InfantAgent-Next

Builds on: GSPO GRPO UI-TARS-1.5 Qwen2.5-VL UGround

Closest papers by meaning

• FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection (CVPR 2026 · similarity 0.92)
• Learning GUI Grounding with Spatial Reasoning from Visual Feedback (ICML 2026 · similarity 0.91)
• Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion (ICML 2026 · similarity 0.90)
• Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.90)
• DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding (CVPR 2026 · similarity 0.90)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.90)