Research map / Multi-agent systems and games / LLM agents
AndroidReality: How Far Are Mobile Agents from the Real World?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AndroidReality, an MDP-grounded perturbation benchmark on AndroidWorld (24 state/action/transition perturbations) that reveals large robustness drops for open-source mobile agents, and a training-free test-time introspective recovery (TTIR) module that reduces failures.
Problem
Mobile GUI agents achieve strong results on clean online benchmarks like AndroidWorld but degrade sharply under real-world interface variability, action execution noise, and unexpected interruptions, which clean benchmarks do not measure.
Approach
Organize real-world mobile perturbations by which MDP component they act on (state, transition, action), inject them as a wrapper into AndroidWorld for fine-grained robustness attribution, and add a training-free diagnose-then-recover loop at test time targeting four identified failure modes.
Key results
- All eight evaluated open-source mobile agents degrade under perturbation, with absolute overall drops of roughly 13 to 36 points (e.g., GUI-Owl-32B 64.66 to 41.81).
- TTIR applied to GUI-Owl-7B reduces failed tasks from 60 to 50 under perturbation (17% relative) and from 44 to 35 on clean tasks (20% relative).
- State-level perturbations cause the most uniform degradation across agents; transition-level perturbations show the largest cross-agent variance (e.g., -2.27 to -29.54 points).
Surprising findings
- Action-level perturbations cause the smallest drops, largely because agents' default retry behavior masks short-lived execution noise.
- TTIR improves performance on the clean setting too, suggesting the failure modes are latent weaknesses rather than perturbation artifacts.
Limitations
- Each task is paired with only one fixed perturbation instance and evaluated once under a fixed seed, limiting statistical power on the small task set.
- Failure tagging relies on manual inspection of trajectories.
- TTIR is evaluated only on GUI-Owl-7B in the main empirical study.
Experimental setup
Datasets: AndroidWorld
Benchmarks: AndroidReality AndroidWorld
Metrics: task success rate success rate drop per perturbation class step overhead
Baselines: GUI-Owl-7B GUI-Owl-32B GUI-Owl-1.5-2B GUI-Owl-1.5-4B GUI-Owl-1.5-8B GUI-Owl-1.5-32B UI-TARS-7B-SFT UI-TARS-1.5-7B
Builds on: AndroidWorld
Closest papers by meaning
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (ICLR 2025 · similarity 0.89)
- SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.88)
- DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections (NeurIPS 2026 · similarity 0.87)
- VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.87)
- MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control (AAAI 2026 · similarity 0.86)
- ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.