Research map / Vision-language and multimodal / Multimodal LLMs

GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GUI-Eyes, an RL-trained Qwen2.5-VL-3B agent that learns when and how to crop or zoom screenshots via a two-stage perception-reasoning loop, using a tool reward combining center proximity and region overlap.

Problem

GUI agents rely on static one-shot screenshots and passive perception, and RL for GUI grounding suffers from sparse rewards and lacks adaptive visual observation.

Approach

Treat visual tool invocation (crop/zoom with learned center and size) as part of the RL policy in a two-stage trajectory, and add a dense spatially continuous tool reward mixing center proximity (Gaussian-like of distance to GT box) and overlap with the GT box.

Key results

• GUI-Eyes-3B achieves 44.8% average accuracy on ScreenSpot-Pro using 3K training samples, outperforming GUI-G1-3B (37.1%) and GUI-R1-3B (30.2%).
• GUI-Eyes-3B reaches 87.8% on ScreenSpot and 88.4% on ScreenSpot-v2.
• Ablation on ScreenSpot-Pro: full tool reward 44.8% vs center-only 39.7% and overlap-only 41.7%; no tool usage gives 30.2%.

Surprising findings

• Fixed static cropping from GUI-R1-3B predictions improves over no tool use but remains below the learned dynamic tool policy.
• Using 3K samples, the model outperforms GUI-G1-3B which was trained on 17K RL samples on ScreenSpot-Pro average (44.8 vs 37.1).

Limitations

• Icon grounding gains are smaller than text gains (e.g., ScreenSpot-Pro icon accuracy 15.7%).
• Reward coefficients and tool hyperparameters tuned via grid search; performance sensitive to reward weighting.
• Evaluation limited to grounding benchmarks; no end-to-end agent task evaluation reported.

Experimental setup

Datasets: OS-Atlas OS-Genesis GUI-R1 AndroidControl ScreenSpot ScreenSpot-v2 ScreenSpot-Pro

Benchmarks: ScreenSpot ScreenSpot-v2 ScreenSpot-Pro

Metrics: grounding accuracy

Baselines: GPT-4o Claude Computer Use Qwen2.5-VL-3B Qwen2.5-VL-7B CogAgent-18B OS-Atlas-7B UI-TARS-7B InfiGUI-R1-3B GUI-R1-3B GUI-G1-3B SE-GUI-3B UI-R1-3B

Builds on: GRPO DeepEyes DeepSeek-R1 Qwen2.5-VL

Closest papers by meaning

• GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.92)
• Learning GUI Grounding with Spatial Reasoning from Visual Feedback (ICML 2026 · similarity 0.92)
• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning (ICLR 2026 · similarity 0.91)
• GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.91)
• GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.90)