Research map / Multi-agent systems and games / LLM agents
SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SPA-Bench, a bilingual (English/Chinese) smartphone agent benchmark with 340 tasks, a plug-and-play Android framework integrating 11 agents, and an automated hybrid action/state success-detection pipeline with seven metrics.
Problem
Fairly comparing (M)LLM-based smartphone agents requires diverse tasks (including Chinese and third-party apps), many agents under one framework, and automated success detection that does not need handcrafted validation logic or human review.
Approach
A unified plug-and-play emulator framework with snapshot-restored Android emulators, plus a coarse-to-fine hybrid (key-component matching followed by GPT-4o judging) success detector for single-app tasks and a subtask-segmentation detector for cross-app tasks, evaluated with seven completion and consumption metrics.
Key results
- M3A achieved the highest single-app success rate of 0.544 overall (0.640 English, 0.447 Chinese) among agentic workflow agents using GPT-4o.
- Agent-as-a-model agents performed poorly, with the best (OdysseyAgent) reaching 0.037 overall single-app success.
- Cross-app success was low: most agents completed at most 4 of 40 cross-app tasks, while M3A completed 6; the success detectors achieved F1 of 0.926 (English) and 0.884 (Chinese) for single-app and 0.845 for cross-app against humans.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an agentic workflow rather than an agent-as-a-model approach | Smartphone agents evaluated on SPA-Bench · SPA-Bench single-app and cross-app tasks | success rate: 0.640 vs 0.053 | Helped |
| Use M3A instead of T3A | Smartphone agents on SPA-Bench · SPA-Bench single-app English | success rate: 0.640 vs 0.487 | Helped |
| Use agentic workflow agents rather than fine-tuned agent-as-a-model agents | Smartphone agents on SPA-Bench · SPA-Bench single-app Chinese | success rate: 0.447 vs 0.027 | Helped |
| Use MobileAgentV2 rather than T3A for Chinese single-app tasks | Smartphone agents on SPA-Bench · SPA-Bench single-app Chinese | success rate: 0.440 vs 0.380 | Mixed |
| Use an agentic workflow rather than an agent-as-a-model approach | Smartphone agents on SPA-Bench · SPA-Bench cross-app tasks | success rate: 0.200 vs 0 | Helped |
| Use the combined actions and shortcuts available to M3A | M3A · SPA-Bench single-app English | mean step ratio on success: 0.92 | Helped |
| Use a coarser a11y-tree input rather than OCR and raw screenshots | T3A versus MobileAgentV2 · SPA-Bench single-app Chinese | success rate: 0.380 vs 0.440 | Hurt |
Surprising findings
- Agents using agentic workflows with fine-tuned models often have high offline action matching scores yet fail in dynamic online settings.
- Agents with lower premature termination rates tended to have higher overdue termination rates (e.g., SeeAct).
Limitations
- Task count is modest (300 single-app and 40 cross-app tasks), and experiments were run once rather than multiple times.
- Single-app success detection relies on GPT-4o and key-component matching; some apps are hard to run in emulators.
- Cost and latency of agentic workflow agents make them impractical for real-world deployment.
Experimental setup
Datasets: GUI Odyssey
Benchmarks: SPA-Bench
Metrics: success rate mean step ratio on success self-reported completion rate maximum steps reached rate error rate premature termination rate overdue termination rate time spent API cost
Baselines: AppAgent AutoDroid MobileAgent MobileAgentV2 M3A T3A SeeAct Auto-UI CogAgent DigiRL OdysseyAgent
Builds on: AppAgent AutoDroid M3A T3A MobileAgent GUI Odyssey
Closest papers by meaning
- FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents (ICLR 2026 · similarity 0.91)
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (ICLR 2025 · similarity 0.90)
- AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (ICML 2026 · similarity 0.90)
- ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices (CVPR 2026 · similarity 0.90)
- VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.89)
- Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.