Research map / Vision-language and multimodal / Multimodal LLMs

GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematic empirical study of rule-based reinforcement fine-tuning (GRPO) for GUI visual grounding, adding a soft reward, point-in-box reward, and an Adversarial KL Factor, yielding GuirlVG on Qwen2.5-VL-7B with only 2K-5.2K training samples.

Problem

GUI visual grounding is usually trained with large-scale SFT, and naive RFT (GRPO) underperforms SFT for this task; the optimal RFT formulation for GUI-VG was unexplored.

Approach

Decompose GRPO for GUI grounding into reward design (soft format reward, point prediction with In-Bbox reward), and introduce an Adversarial KL Factor that scales the KL penalty by reward/maximum reward to counter reward over-optimization, plus tuned training configurations (LoRA, group/batch size, resolution prompting).

Key results

• GuirlVG-7B with 5.2K samples reaches 88.7% on ScreenSpot (+7.7% over OS-Atlas-7B) and 91.9% on ScreenSpot v2.
• On ScreenSpot-Pro, GuirlVG-5.2K-7B reaches 36.1%, +17.2% over OS-Atlas-7B.
• Adversarial KL Factor improves ScreenSpot accuracy from 85.6% to 87.4% over the best fixed beta.

Surprising findings

• Naive/trivial GRPO-based RFT underperforms the SFT baseline (79.2% vs 82.6% on ScreenSpot at 500 steps).
• Withholding image resolution during training but adding it at test time gives the best accuracy (88.0%), while providing it in training hurts (83.7%).

Limitations

• Only evaluated Qwen2.5-VL; other MLLMs such as Qwen2-VL not explored.
• Larger models (32B, 72B) not explored due to compute constraints.
• Limited exploration of alternative training datasets.

Experimental setup

Datasets: ShowUI OmniAct

Benchmarks: ScreenSpot ScreenSpot v2 ScreenSpot-Pro

Metrics: accuracy (point inside ground-truth bbox)

Baselines: SFT baseline (Qwen2.5-VL, LLaMA Factory) SeeClick CogAgent AGG OmniParser UGround ShowUI-G ShowUI OS-Atlas-4B OS-Atlas-7B Fuyu

Builds on: GRPO DeepSeek-R1 Qwen2.5-VL LoRA LLaMA Factory

Closest papers by meaning

• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.93)
• GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.92)
• GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.91)
• Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.91)
• Test-Time Reinforcement Learning for GUI Grounding via Region Consistency (AAAI 2026 · similarity 0.91)
• GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.91)