Research map / Multi-agent systems and games / LLM agents

VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics

ICML 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VenusBench-Mobile, a user-intent-driven online benchmark of 149 primary tasks plus 80 stability variants for mobile GUI agents, with a five-dimension PUDAM capability diagnostic, and evaluates SOTA agents showing large drops.

Problem

Existing online mobile GUI benchmarks are app-centric and task-homogeneous, and report coarse success rates that do not diagnose which capabilities cause failures or test robustness to environment variations.

Approach

Build tasks top-down from ten user-intent categories (e.g., function assistance, conflict, vague instructions, multi-round, noise resistance) rather than per app, annotate each task with a five-dimension capability taxonomy (PUDAM) at four levels, and add environment-variant stability tests measured by an all-variants pass rate.

Key results

• Gemini-3-Pro + UI-Venus-72B achieves the highest total success rate of 36.9% on VenusBench-Mobile, while most open-source agents stay below 15%.
• Average SOTA success drops by about 50 points relative to AndroidWorld, per the paper's comparison.
• Stability Pass Rate is 0% for most models and at most 15% (Gemini-3-Pro + UI-Venus-72B), with Pad (tablet) layout causing the largest drops.

Surprising findings

• Memory is the absolute bottleneck across models, with open-source models dropping to near zero at advanced memory levels.
• The multi-agent Mobile-Agent-v3 framework consumes about 438.7 tokens per step, far more than its base GUI-Owl-7B model at 146.5, without a clear accuracy gain.

Limitations

• The primary pool is small at 149 tasks across 27 apps, and some capability levels have no test cases.
• Verification for 90 tasks relies on an MLLM judge (Qwen3-VL-30B-A3B-Instruct) rather than programmatic checks.

Experimental setup

Datasets: AndroidWorld tasks (adapted for noise resistance)

Benchmarks: VenusBench-Mobile AndroidWorld

Metrics: success rate dimension-wise accuracy stability pass rate total tokens per-step tokens

Baselines: AndroidWorld Qwen3-VL-30B-A3B Qwen3-VL-8B UI-Venus-72B UI-Venus-7B GUI-Owl-7B Mobile-Agent-v3 MAI-UI-8B Gemini-3-Pro + UI-Venus-72B GPT-5.1 + UI-Venus-72B

Builds on: AndroidWorld BrowseComp

Closest papers by meaning

• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.92)
• GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents (CVPR 2026 · similarity 0.91)
• ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.90)
• Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · similarity 0.90)
• SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.89)
• FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents (ICLR 2026 · similarity 0.89)