Research map / Vision-language and multimodal / Multimodal LLMs

GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes K-step GUI Transition, a self-supervised inverse-dynamics task where VLMs predict the first action between two GUI screenshots, and GUI-Shift, which trains it with GRPO and rule-based rewards on 2K filtered AndroidControl-derived samples.

Problem

Training VLM-based GUI agents typically needs large human-annotated instruction datasets for trajectories, which are costly and error-prone; the paper asks how to train capable mobile GUI agents from unlabeled GUI trajectories.

Approach

Replace textual instructions with the target screenshot S_{t+k} as a visual goal, train the VLM to predict the first action linking S_t to S_{t+k}, and optimize with GRPO using rule-based format and action-correctness rewards (click correct if inside the target bounding box), with filtering to keep samples having both correct and incorrect sampled responses.

Key results

• GUI-Shift-Qwen (k=1) reaches 70.4% EM on AndroidControl-High, up to 11.2 points above Qwen2.5-VL-7B base.
• GUI-Shift-Qwen (k=4) reaches 89.0% on ScreenSpot-v2, +1.3 over base; GUI-Shift-Mimo-RL gains 1.5 on ScreenSpot-Pro.
• GRPO outperforms SFT on K-step GUI Transition; removing reasoning halves training time (17 to 9 hours for Qwen2.5-VL-7B).

Surprising findings

• SFT on the same task reduced accuracy relative to both base models and GRPO, with drops up to 65.1% relative to GRPO.
• Training with GUI-Shift lowered GUI Odyssey scores for MiMo-VL variants, attributed to tablet episodes in the test set.

Limitations

• Gains on GUI Odyssey are negative for some MiMo-VL variants, notably on tablet layouts.
• ScreenSpot-Pro gains are small and some configurations decline, attributed to high-resolution desktop screenshots absent from training data.
• Data filtering results are mixed across models and k values.

Experimental setup

Datasets: AndroidControl

Benchmarks: AndroidControl GUI Odyssey ScreenSpot-v2 ScreenSpot-Pro

Metrics: type match exact match accuracy

Baselines: GPT-4o SeeClick OS-Atlas-7B Aguvis-7B UI-TARS-7B UI-R1-3B GUI-R1-7B InfiGUI-R1-3B AgentCPM-GUI UI-Venus-Navi-7B UI-Venus-Ground-7B UGround-7B

Builds on: GRPO DeepSeek-R1 VLM-R1

Closest papers by meaning

• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
• GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.91)
• GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning (ICLR 2026 · similarity 0.91)
• UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents (NeurIPS 2025 · similarity 0.90)
• GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding (NeurIPS 2026 · similarity 0.90)