AI Paper Atlas

Research map / Multi-agent systems and games / LLM agents

SMAN-Bench: A Cross-System Benchmark for Mobile Agents under Single- and Multi-path, Ambiguous, and Noisy Tasks

Weikai Xu, Zhizheng Jiang, Yuxuan Liu, Pengzhi Gao, Wei Liu, Jian Luan, Yunxin Liu, Yuanchun Li, Bin Wang, Bo An

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SMAN-Bench, a cross-system benchmark for VLM-based mobile agents covering single- and multi-path, ambiguous, and noisy tasks, aimed at overcoming the unstable reward signals of online benchmarks and the single-path limitation of offline benchmarks.

Problem

Existing online mobile-agent benchmarks fail to give stable critical reward signals under dynamic environment changes and ignore noise and interactive instructions; offline benchmarks evaluate only single-path trajectories.

Approach

A benchmark that evaluates mobile agents across single- and multi-path tasks, ambiguous instructions, and noisy environments, spanning multiple systems, to address the reward-stability and single-path limitations of prior benchmarks.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use multi-path instead of single-path evaluationagent frameworks · SMAN-Bench Commonsuccess rate: MobileAgent-E GPT-4o simple SR 27.5 (single) -> 30.5 (multi)Improved
add ads and pop-ups noise to tasksVLM agents · SMAN-Bench Noisysuccess rate: Claude 4.5 Sonnet 39.0 -> 15.5Hurt
insert ads into AITZ trajectoriesQwen2-VL-7B and OS-Atlas-7B · AITZ-Noisetotal step accuracy: 46.9 -> 43.9 and 48.6 -> 45.1Hurt
add in-domain noisy data to fine-tuning (ablation)Qwen2-VL-7B · AITZ-Noisenoisy step accuracy: 17.46 -> 75.79Improved
add out-domain noisy data to fine-tuning (ablation)OS-Atlas-7B · AITZ-Noisenoisy step accuracy: 21.62 -> 55.47Improved
replace full instruction with ambiguous instruction plus step-by-step Q&A (ablation)AppAgent with Llama3.2-VL-90B · SMAN-Bench Ambiguousstep accuracy: 7.6 -> 15.5Improved
replace full instruction with ambiguous instruction plus Q&A (ablation)AppAgent with GPT-4o · SMAN-Bench Ambiguousstep accuracy: 31.9 -> 33.9No clear effect
use MobileAgent-E framework instead of AppAgent-v1general VLMs · SMAN-Bench Commonsuccess rate: Qwen-VL-Max simple SR 20.5 -> 32.5Improved
place thinking models inside agent frameworksreasoning VLMs · SMAN-Bench Commonsuccess rateNo clear effect
use RL-trained small agent versus pretrained larger agentGUI-G1-3B vs OS-Atlas-7B-pro · SMAN-Bench Common-Simplesuccess rate: 25.5 vs 28.0No clear effect

Experimental setup

Benchmarks: SMAN-Bench

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy