Research map / Multi-agent systems and games / LLM agents
SMAN-Bench: A Cross-System Benchmark for Mobile Agents under Single- and Multi-path, Ambiguous, and Noisy Tasks
Weikai Xu, Zhizheng Jiang, Yuxuan Liu, Pengzhi Gao, Wei Liu, Jian Luan, Yunxin Liu, Yuanchun Li, Bin Wang, Bo An
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SMAN-Bench, a cross-system benchmark for VLM-based mobile agents covering single- and multi-path, ambiguous, and noisy tasks, aimed at overcoming the unstable reward signals of online benchmarks and the single-path limitation of offline benchmarks.
Problem
Existing online mobile-agent benchmarks fail to give stable critical reward signals under dynamic environment changes and ignore noise and interactive instructions; offline benchmarks evaluate only single-path trajectories.
Approach
A benchmark that evaluates mobile agents across single- and multi-path tasks, ambiguous instructions, and noisy environments, spanning multiple systems, to address the reward-stability and single-path limitations of prior benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-path instead of single-path evaluation | agent frameworks · SMAN-Bench Common | success rate: MobileAgent-E GPT-4o simple SR 27.5 (single) -> 30.5 (multi) | Improved |
| add ads and pop-ups noise to tasks | VLM agents · SMAN-Bench Noisy | success rate: Claude 4.5 Sonnet 39.0 -> 15.5 | Hurt |
| insert ads into AITZ trajectories | Qwen2-VL-7B and OS-Atlas-7B · AITZ-Noise | total step accuracy: 46.9 -> 43.9 and 48.6 -> 45.1 | Hurt |
| add in-domain noisy data to fine-tuning (ablation) | Qwen2-VL-7B · AITZ-Noise | noisy step accuracy: 17.46 -> 75.79 | Improved |
| add out-domain noisy data to fine-tuning (ablation) | OS-Atlas-7B · AITZ-Noise | noisy step accuracy: 21.62 -> 55.47 | Improved |
| replace full instruction with ambiguous instruction plus step-by-step Q&A (ablation) | AppAgent with Llama3.2-VL-90B · SMAN-Bench Ambiguous | step accuracy: 7.6 -> 15.5 | Improved |
| replace full instruction with ambiguous instruction plus Q&A (ablation) | AppAgent with GPT-4o · SMAN-Bench Ambiguous | step accuracy: 31.9 -> 33.9 | No clear effect |
| use MobileAgent-E framework instead of AppAgent-v1 | general VLMs · SMAN-Bench Common | success rate: Qwen-VL-Max simple SR 20.5 -> 32.5 | Improved |
| place thinking models inside agent frameworks | reasoning VLMs · SMAN-Bench Common | success rate | No clear effect |
| use RL-trained small agent versus pretrained larger agent | GUI-G1-3B vs OS-Atlas-7B-pro · SMAN-Bench Common-Simple | success rate: 25.5 vs 28.0 | No clear effect |
Experimental setup
Benchmarks: SMAN-Bench
Closest papers by meaning
- SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.91)
- VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.91)
- ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices (CVPR 2026 · similarity 0.90)
- FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents (ICLR 2026 · similarity 0.89)
- MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning (ICLR 2026 · similarity 0.89)
- Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automation (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
