Research map / Vision-language and multimodal / Multimodal LLMs

Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HyperTrack, a Chinese mobile GUI dataset (~16k tasks, 674 apps), and GUIEvalKit, an offline/semi-online evaluation toolkit, then studies SFT vs DAPO-style RL data scaling for UI-TARS-1.5-7B and the effect of reasoning on GUI agent decisions.

Problem

Mobile GUI navigation agents lack large-scale Chinese-app training/evaluation data, standardized benchmarking tools, a clear understanding of how data scale and RL affect generalization, and how explicit reasoning affects task completion.

Approach

Scales SFT vs. DAPO-style RL over 16 to 8192 episodes, finds log-linear scaling with RL generalizing better out-of-domain, proposes semi-online evaluation that injects the model's own correct history, and introduces decision diversity/stability metrics to analyze reasoning trade-offs.

Key results

• RL (DAPO-style GRPO) consistently outperforms SFT at matched data scales, with the gap largest on out-of-domain unseen-app splits; performance scales approximately linearly with log of training episodes.
• Thinking mode often hurts offline step match: e.g., Qwen3-VL-8B-Instruct 81.48/66.28 vs Thinking 77.03/59.35 on HyperTrack.
• SOEval raises average step exact match across five benchmarks (e.g., Qwen3-VL-4B-Instruct 59.39 to 63.05; UI-Venus-Navi-72B 74.09 to 76.16) and correlates better with AndroidWorld online success than offline metrics.

Surprising findings

• Thinking mode frequently degrades GUI navigation performance, with instruct variants outperforming thinking variants at 4B and 8B scales.
• Reasoning underperforms instruct-only at pass@1 but surpasses it at pass@8, reflecting a stability-diversity trade-off.

Limitations

• Full HyperTrack dataset is not publicly released; only a preview test subset is available pending review.
• SOEval is a step-level approximation and does not substitute for full online evaluation; correlation with AndroidWorld is moderate.
• Decision-level metrics depend on DBSCAN clustering with a fixed epsilon, which is a heuristic choice.

Experimental setup

Datasets: HyperTrack AndroidControl AiTZ GUI Odyssey CAGUI

Benchmarks: AndroidControl GUI-Odyssey AiTZ CAGUI HyperTrack AndroidWorld

Metrics: step type match step exact match episode progress episode success rate pass@n decision diversity decision stability

Baselines: UI-TARS-7B-SFT UI-TARS-72B-DPO AgentCPM-GUI-8B MagicGUI-RFT UI-Venus-Navi-72B GUI-Owl-32B MiMo-VL-7B-RL-2508 Qwen3-VL-8B-Instruct

Builds on: GRPO DAPO UI-TARS vLLM

Closest papers by meaning

• GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents (CVPR 2026 · similarity 0.90)
• GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.89)
• GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation (NeurIPS 2025 · similarity 0.88)
• Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.88)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.88)