Research map / Vision-language and multimodal / Multimodal LLMs
GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GUI-Reflection, a framework that injects self-reflection and error-correction into end-to-end mobile GUI models across GUI pre-training, offline SFT with automatically generated reflection data, and iterative online reflection tuning in a new Android environment.
Problem
End-to-end multimodal GUI models trained on near error-free offline trajectories lack reflection and error recovery abilities, and offline SFT can erode the base MLLM's reflection behavior.
Approach
Decomposes reflection into action verification, action reversal, and mistake-informed reattempt tasks; automatically synthesizes reflection/correction data from successful trajectories with MLLMs; and runs an iterative online tuning loop that annotates pre-error corrections and post-error reflections for failed rollouts, without human annotation.
Key results
- On Action Verification, GUI-Pretrain-Ref-8B reaches 87.56% accuracy versus 57.95% for GUI-Pretrain-8B, on par with Gemini-2.5-Pro (88.22%).
- Online reflection tuning raises level-2 success in the custom environment from 14.58% (filtered BC baseline) to 34.72%, with offline reflection data alone giving 23.61%.
- Achieves 34.5% success on AndroidWorld among end-to-end models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GUI-Reflection Task Suite training data during GUI pre-training (ablation) | GUI-Pretrain · Action Verification | accuracy: 57.95 -> 87.56 | Helped |
| Add GUI-Reflection Task Suite training data during GUI pre-training (ablation) | GUI-Pretrain · Action Reversal | accuracy: 40.71 -> 93.81 | Helped |
| Add GUI-Reflection Task Suite training data during GUI pre-training (ablation) | GUI-Pretrain · ScreenSpot | average score: 83.58 -> 85.12 | Helped |
| Add GUI-Reflection Task Suite training data during GUI pre-training (ablation) | GUI-Pretrain · ScreenSpotv2 | average score: 84.84 -> 86.88 | Helped |
| Add reflection data during offline SFT (ablation) | GUI-Pretrain-Ref with filtered BC · GUI environment level-2 tasks | success rate: 14.58\% -> 23.61\% | Helped |
| Use online reflection tuning instead of filtered BC (ablation) | Offline reflection SFT model · GUI environment level-2 tasks | success rate: 23.61\% -> 34.72\% | Helped |
| Provide a second mistake-informed grounding attempt (ablation) | GUI-Pretrain-Ref · ScreenSpot | average score: 85.12 -> 88.00 | Helped |
| Provide a second mistake-informed grounding attempt (ablation) | GUI-Pretrain-Ref · ScreenSpotv2 | average score: 86.88 -> 89.27 | Helped |
| Use the 3rd mistake-informed attempt instead of the initial prediction (ablation) | GUI-Pretrain-Ref · ScreenSpot | average score: 85.12 -> 89.61 | Helped |
| Use the 3rd mistake-informed attempt instead of the initial prediction (ablation) | GUI-Pretrain-Ref · ScreenSpotv2 | average score: 86.88 -> 90.50 | Helped |
| Use pass@3 rather than the sequential 3rd attempt (ablation) | GUI-Pretrain-Ref · ScreenSpot | average score: 89.61 -> 87.35 | Hurt |
| Use pass@3 rather than the sequential 3rd attempt (ablation) | GUI-Pretrain-Ref · ScreenSpotv2 | average score: 90.50 -> 88.84 | Hurt |
Surprising findings
- GUI pre-training degrades the base MLLM's reflection-related abilities: GUI-Pretrain-8B is far below InternVL3-8B on action verification negatives (TN 21.55 vs 26.86) and reversal (40.71 vs 63.80).
- Mistake-informed reattempt with 3rd attempt beats pass@3 only after adding reflection data to pre-training (GUI-Pretrain-Ref), not for the base or plain GUI-pretrained models.
Limitations
- Reflection data focuses on visual and action-grounded errors, not deeper high-level planning errors.
- Only evaluated on mobile environments; desktop and web need new data and engineering.
- Relies on MLLM annotators (Gemini, GPT-4o) for generated reflection and verification labels.
Experimental setup
Datasets: AndroidControl GUI-Odyssey AITW AITZ AMEX Wave-UI OS-ATLAS-Desktop UI RefExp Widget Captioning SeeClick-Rico RICO Semantics OpenApp
Benchmarks: GUI-Reflection Task Suite (Action Verification, Action Reversal) ScreenSpot ScreenSpotv2 AndroidWorld custom online environment level-2 tasks
Metrics: accuracy TN TP success rate pass@3
Baselines: GPT-4o Gemini-2.5-Pro Gemini-2.5-Flash Claude-3.7-Sonnet Qwen2.5-VL-72B InternVL3-78B InternVL3-8B InternVL2.5-8B GUI-Pretrain filtered BC
Builds on: InternVL2.5 UI-TARS AndroidWorld filtered behavior cloning
Closest papers by meaning
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.89)
- Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation (NeurIPS 2025 · similarity 0.88)
- SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.87)
- WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments (ICML 2026 · similarity 0.87)
- Reflection with Action-Induced Visual Differences for Desktop GUI Agents (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.