Research map / Large language models / Reasoning and chain of thought

Test-Time Reinforcement Learning for GUI Grounding via Region Consistency

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GUI-RC, a test-time voting method over multiple sampled GUI grounding predictions, and GUI-RCPO, which turns region-consistency rewards into GRPO-based test-time RL on unlabeled ScreenSpot-v2 data, improving several VLMs on ScreenSpot benchmarks.

Problem

GUI grounding (mapping natural language instructions to screen coordinates) relies on costly pixel-level labels and train-time scaling, leaving test-time computation underused.

Approach

Sample multiple predictions, build a spatial voting grid, and take the largest maximum-vote region as the consensus; the same vote-density measure is used as a label-free reward for GRPO-based test-time RL.

Key results

• GUI-RC improves OS-Atlas-Base-7B by 2.75% on SSv2 average (80.82 to 83.57).
• GUI-RCPO raises Qwen2.5-VL-3B-Instruct SSv2 average from 80.11 to 85.14 using 1,272 unlabeled samples.
• GUI-RCPO on Qwen2.5-VL-7B-Instruct improves SSPro average from 19.80 to 25.93.

Surprising findings

• GUI-RCPO exceeds GUI-RC even though its reward is derived from the GUI-RC consensus, which the authors attribute to learning a better grounding strategy rather than fitting the consensus.
• GUI-RC gives limited gains for point-style models (e.g., InternVL3-8B drops on SSPro, 13.28 to 12.46).

Limitations

• Gains are limited for point-style grounding models because the point-to-box expansion with alpha is a heuristic approximation.
• Cannot fix confusion hallucinations where the model fails to recognize any valid UI element; requires the base model to be familiar with GUI environments.

Experimental setup

Datasets: ScreenSpot-v2

Benchmarks: ScreenSpot ScreenSpot-v2 ScreenSpot-Pro

Metrics: grounding accuracy

Baselines: greedy decoding InternVL3-2B-Instruct InternVL3-8B-Instruct Qwen2.5-VL-3B-Instruct Qwen2.5-VL-7B-Instruct UGround-V1-7B UI-TARS-1.5-7B OS-Atlas-Base-7B GUI-G2-7B

Builds on: self-consistency TTRL GRPO VLM-R1

Closest papers by meaning

• GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents (NeurIPS 2025 · similarity 0.92)
• Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.91)
• GTA1: GUI Test-time Scaling Agent (ICLR 2026 · similarity 0.91)
• GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning (ICLR 2026 · similarity 0.91)
• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Learning GUI Grounding with Spatial Reasoning from Visual Feedback (ICML 2026 · similarity 0.90)