Research map / Multi-agent systems and games / LLM agents
UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UI-Genie, a self-improving framework that pairs an image-text interleaved reward model (UI-Genie-RM) with reward-guided exploration to iteratively generate GUI agent training trajectories without manual annotation.
Problem
Mobile GUI agents lack reliable trajectory outcome verification and scalable high-quality training data, since current training relies on human-annotated trajectories and proprietary judges.
Approach
A reward model that takes interleaved recent screenshots and action summaries as context and unifies action-level and task-level rewards by treating "task completion" as a special action; it guides beam-search trajectory exploration, and both agent and reward model are iteratively retrained over three rounds of increasing task complexity.
Key results
- UI-Genie-Agent-3B reaches 93.8% low-level and 72.9% high-level SR on AndroidControl, and the 72B variant reaches 77.0% high-level SR, exceeding UI-TARS by 2.3%.
- UI-Genie-Agent-7B on AndroidLab rises from 18.1% (round 0) to 38.7% SR (round 3), surpassing AutoGLM's 36.2%.
- UI-Genie-RM achieves 79.6% step-level and 82.1% outcome-level F1 overall, outperforming GPT-4o (68.1 / 66.7) and Gemini2.5-pro (72.9 / 74.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train UI-Genie-Agent with the UI-Genie synthetic-data and self-improvement framework | Qwen2.5-VL-based mobile GUI agents · AndroidLab | Success Rate: 18.1% -> 38.7% | Helped |
| Use UI-Genie-RM for reward-guided test-time selection (ablation) | Qwen2.5-VL-3B · AndroidControl | Step SR: 60.3% -> 63.1% | Helped |
| Use UI-Genie-RM for reward-guided test-time selection (ablation) | Qwen2.5-VL-3B · AndroidControl | Task SR: 7.5% -> 10.2% | Helped |
| Use UI-Genie-RM for reward-guided test-time selection (ablation) | Qwen2.5-VL-7B · AndroidControl, medium tasks | Task SR: 10.5% -> 12.2% | Helped |
| Remove historical information and use only the current screenshot (ablation) | UI-Genie-RM · Reward evaluation, step-level | Performance: 79.6% -> 61.3% | Hurt |
| Remove historical information and use only the current screenshot (ablation) | UI-Genie-RM · Reward evaluation, outcome-level | Performance: 82.1% -> 60.5% | Hurt |
| Increase historical screenshot context from one image to five (ablation) | UI-Genie-RM · Reward evaluation, step-level | Performance: 72.4% -> 79.6% | Helped |
| Increase historical screenshot context from one image to five (ablation) | UI-Genie-RM · Reward evaluation, outcome-level | Performance: 73.8% -> 82.1% | Helped |
| Train separate step-level and task-level reward models instead of a unified model (ablation) | UI-Genie-RM · Reward evaluation | Performance | Hurt |
| Increase best-of-N candidate sampling from N=5 to N=10 (ablation) | Qwen2.5-VL-3B with UI-Genie-RM · AndroidControl | Task SR: 8.3 -> 10.2 | Helped |
| Increase best-of-N candidate sampling from N=5 to N=10 (ablation) | Qwen2.5-VL-7B with UI-Genie-RM · AndroidControl | Task SR: 15.1 -> 16.2 | Helped |
| Refine UI-Genie-RM through self-improvement rounds (ablation) | UI-Genie-RM · Reward benchmark, step-level | Accuracy: 68.2% -> 79.6% | Helped |
Surprising findings
- Off-the-shelf proprietary MLLMs such as GPT-4o and Gemini 2.5 Pro perform only moderately as GUI reward judges (F1 about 65 to 74), even with visual prompts.
Limitations
- Reward model can produce suboptimal rewards, leading to failed trajectories during data expansion.
- Framework cannot guarantee fully correct synthetic trajectories across all GUI tasks.
- Hard tasks (more than 10 steps) still show low task success rates, e.g. around 1-2% for 3B and 7B on A3 and AndroidControl hard subsets.
Experimental setup
Datasets: AndroidControl AMEX AndroidLab UI elements annotations (unnamed source)
Benchmarks: AndroidControl AndroidLab Android Agent Arena (A3) UI-Genie-RM custom reward benchmark (1,050 items)
Metrics: Success Rate Type accuracy Grounding accuracy Sub-Goal Success Rate Reversed Redundancy Ratio Reasonable Operation Ratio Func SR LLM SR ESAR F1-score
Baselines: GPT-4o Gemini1.5-pro Gemini2.5-pro Qwen2.5-VL-7B Qwen2.5-VL-72B UI-TARS OS-Atlas InfiGUI-R1 AutoGLM Aria-UI
Builds on: Qwen2.5-VL Math-Shepherd-style potential-based process labeling AndroidControl AndroidLab AMEX
Closest papers by meaning
- GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.90)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.90)
- HATS: Hardness-Aware Trajectory Synthesis for GUI Agents (CVPR 2026 · similarity 0.88)
- VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.88)
- Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation (ICML 2026 · similarity 0.88)
- SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.