Research map / Large language models / Agents, code and math
GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GUI-Rise, a GUI navigation agent that couples structured CoT reasoning (progress estimation and decision reasoning) with compact textual history summaries, trained by SFT on GPT-4o-mini pseudo-labeled trajectories followed by GRPO with a history-summary reward.
Problem
Cross-domain generalization and effective use of interaction history are limited for MLLM-based GUI navigation agents, which either rely on frozen proprietary LLMs, overfit SFT on static pairs, or encode history as action-only sequences or costly screenshots.
Approach
Integrate structured CoT (progress estimation, decision reasoning), action prediction, and LLM-generated textual history summaries into one output, and reward summaries by whether the model's next-step action derived from them is correct (history summary reward inside GRPO).
Key results
- Qwen2.5-VL-3B-based GUI-Rise reaches 47.6 Step SR on Mind2Web cross-domain in the zero-shot setting versus 24.8 for the base Qwen2.5-VL-3B-zero-shot row... (zero-shot cross-domain Step SR 30.1 vs 22.8 base)
- On AITW in-domain, GUI-Rise with Qwen2-VL-2B achieves 71.1 overall vs 70.0 for ShowUI-2B.
- On AITW zero-shot, GUI-Rise (Qwen2-VL-2B) reaches 54.1 overall vs 35.9 for ShowUI-2B, a relative gain of about 50.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GUI-Rise instead of ShowUI-2B | Qwen2-VL-2B on Mind2Web · Mind2Web cross-domain, zero-shot | Step SR: 21.4 -> 29.7 | Helped |
| Use GUI-Rise instead of Qwen2.5-VL-3B | Qwen2.5-VL-3B on Mind2Web · Mind2Web cross-domain, standard | Step SR: 44.1 -> 47.6 | Helped |
| Use GUI-Rise instead of ShowUI-2B | Qwen2-VL-2B on AITW · AITW zero-shot | Overall action accuracy: 35.9 -> 54.1 | Helped |
| Use GUI-Rise instead of ShowUI-2B | Qwen2-VL-2B on AITW · AITW in-domain | Overall action accuracy: 70.0 -> 71.1 | Helped |
| Use GUI-Rise-2B instead of ShowUI-2B | MiniWob · MiniWob zero-shot | success rate: 27.1 -> 30.6 | Helped |
| Add TST without SCoT, HS, or HSR (ablation) | AITW ablation baseline · AITW in-domain ablation | Overall action accuracy: 67.2 -> 66.0 | Hurt |
| Add SCoT without TST, HS, or HSR (ablation) | AITW ablation baseline · AITW in-domain ablation | Overall action accuracy: 67.2 -> 42.6 | Hurt |
| Add TST and SCoT (ablation) | AITW ablation baseline · AITW in-domain ablation | Overall action accuracy: 67.2 -> 69.8 | Helped |
| Add history summaries (HS) (ablation) | AITW with TST and SCoT · AITW in-domain ablation | Overall action accuracy: 69.8 -> 70.7 | Helped |
| Add history summary reward (HSR) (ablation) | AITW with TST, SCoT, and HS · AITW in-domain ablation | Overall action accuracy: 70.7 -> 71.1 | Helped |
Surprising findings
- SFT alone on structured CoT drops success substantially (AITW overall 42.6 vs 67.2 without CoT), indicating SFT struggles to learn structured reasoning.
- Position reward stays near zero for RL-only training on Mind2Web, so cold-start SFT is required for meaningful advantage signals.
Limitations
- Trained entirely offline and cannot adapt to new scenarios from live interaction.
- Pseudo-labels depend on GPT-4o-mini quality, and cold-start data is small and synthetic.
- Evaluations use small 2B-3B models in the main results; 7B results are limited to SFT.
Experimental setup
Datasets: Mind2Web AITW GUIAct MiniWob
Benchmarks: Mind2Web AITW MiniWob Android World OSWorld
Metrics: Ele.Acc OP.F1 Step SR action accuracy success rate
Baselines: MindAct GPT-4 OmniParser CogAgent SeeClick Qwen2-VL-2B ShowUI-2B Qwen2.5-VL-3B UI-Tars-2B Infigui-2B
Builds on: GRPO PPO ShowUI UI-R1 Qwen2-VL
Closest papers by meaning
- History-Aware Reasoning for GUI Agents (AAAI 2026 · similarity 0.90)
- HiconAgent: History Context-aware Policy Optimization for GUI Agents (CVPR 2026 · similarity 0.89)
- Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation (ICML 2026 · similarity 0.88)
- GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.87)
- UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction as Reasoning (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.