Research map / Multi-agent systems and games / LLM agents
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AndroidWorld, a dynamic, reproducible Android environment with 116 parameterized tasks across 20 apps (plus 92 MobileMiniWoB++ tasks) and a multimodal agent M3A, reporting that the best agent completes 30.6% of tasks versus 80% for humans.
Problem
Evaluating autonomous computer-control agents on mobile devices lacks realistic, reproducible benchmarks with automated outcome-based rewards; existing mobile benchmarks are small or use static test specifications.
Approach
Use Android system state (adb-accessible databases, files, settings) as durable reward signals and randomly parameterize tasks with seeds, so each task has initialization, success checking, and teardown logic that remains valid across many task instantiations.
Key results
- M3A with GPT-4 Turbo (accessibility tree) reaches 30.6% success on AndroidWorld, versus 80.0% for humans and 15.5% for the adapted SeeAct.
- Across three random seeds, M3A with GPT-4 Turbo scores 27.6%, 26.3% and 33.2% (mean 29.0%), showing substantial variance from task parameters.
- On a Pixel 5 (Android 12) the agent achieves 28.4% success, close to the primary setup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add M3A's guidelines and reflection mechanisms compared with M3A-Simple (ablation) | M3A with GPT-4 Turbo on AndroidWorld · AndroidWorld | success rate: 19.8% vs 30.6% | Helped |
| Add M3A's guidelines and reflection mechanisms compared with M3A-Simple (ablation) | M3A with GPT-4 Turbo on MobileMiniWoB++ · MobileMiniWoB++ | success rate: 67.7% | No clear effect |
| Use SoM screen input in addition to the accessibility tree instead of accessibility-tree-only input (ablation) | M3A with GPT-4 Turbo · AndroidWorld | success rate: 30.6 -> 25.4 | Hurt |
| Use SoM screen input in addition to the accessibility tree instead of accessibility-tree-only input (ablation) | M3A with GPT-4 Turbo · MobileMiniWoB++ | success rate: 59.7 -> 67.7 | Helped |
| Use SoM screen input in addition to the accessibility tree instead of accessibility-tree-only input (ablation) | M3A with Gemini 1.5 Pro · AndroidWorld | success rate: 19.4 -> 22.8 | Helped |
| Use SoM screen input in addition to the accessibility tree instead of accessibility-tree-only input (ablation) | M3A with Gemini 1.5 Pro · MobileMiniWoB++ | success rate: 57.4 -> 40.3 | Hurt |
| Evaluate M3A across different task-parameter seeds rather than reporting only a single seed | M3A with GPT-4 Turbo and accessibility-tree input · AndroidWorld | success rate: 27.6%, 26.3% and 33.2% (mean 29.0%) | Mixed |
| Change device and OS from Pixel 6 (Android 13) to Pixel 5 (Android 12) | M3A · AndroidWorld | success rate: 28.4% | Mixed |
| Use M3A instead of SeeAct | GPT-4 Turbo agent on AndroidWorld · AndroidWorld | success rate: 15.5 -> 30.6 | Helped |
Surprising findings
- Adding screenshots with Set-of-Mark annotations lowered AndroidWorld success (25.4% vs 30.6% text-only with GPT-4 Turbo), though it helped on MobileMiniWoB++.
- Performance varied even with a fixed seed, indicating model non-determinism affects reliability.
Limitations
- Only open-source and built-in apps are supported; trending popular apps are not covered.
- Several MiniWoB++ tasks were excluded because they failed on Android, leaving 92 tasks.
- Human baseline based on only two participants with one attempt per task.
Experimental setup
Datasets: AndroidWorld tasks MiniWoB++
Benchmarks: AndroidWorld MobileMiniWoB++
Metrics: success rate
Baselines: SeeAct M3A-Simple Human
Builds on: ReAct Reflexion SeeAct Set-of-Mark AndroidEnv MiniWoB++
Closest papers by meaning
- SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION (ICLR 2025 · similarity 0.90)
- AndroidReality: How Far Are Mobile Agents from the Real World? (NeurIPS 2026 · similarity 0.89)
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.89)
- AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (ICML 2026 · similarity 0.89)
- Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · similarity 0.89)
- On the Effects of Data Scale on UI Control Agents (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.