Research map / Multi-agent systems and games / LLM agents
DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DistRL, an asynchronous distributed RL framework (centralized host learner with decentralized Android workers) and the A-RIDE off-policy algorithm to fine-tune a 1.3B T5-based multimodal agent online, improving training efficiency and success on AitW General and Web Shopping tasks.
Problem
Online RL fine-tuning of MLLM-based on-device (mobile Android) control agents is inefficient because data collection in parallel emulators/devices is synchronous and task durations vary widely, and off-policy asynchronous data creates non-stationarity and stale-policy issues.
Approach
Decouple data collection (heterogeneous worker devices running asynchronous threads) from centralized GPU policy training, and train with A-RIDE, an advantage-based Retrace off-policy correction combined with distributed prioritized experience replay that prioritizes trajectories by TD error, importance ratio and entropy, plus entropy and invalid-action penalties; rewards come from a Gemini-1.5-Pr
Key results
- DistRL reaches 73.2% success on AitW General test versus 61.2% for DigiRL (multi), about 19.6% relative improvement.
- DistRL achieves 68.5% success on AitW Web Shopping test versus 59.9% for DigiRL (multi).
- DistRL collects 800 trajectories in 6k seconds versus 300 for multi-machine DigiRL, and scales to ~7.7 trajectories per minute on 192 vCPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DistRL instead of DigiRL (multi) | T5-based on-device control agent · AitW General test | success rate: 73.2 \pm 1.1 vs. 61.2\pm2.4; approximately 19.6% relative improvement | Helped |
| use DistRL instead of DigiRL (multi) | T5-based on-device control agent · AitW Web Shopping test | success rate: 68.5 \pm 1.7 vs. 59.9\pm2.8; about 14.4% | Helped |
| use DistRL's distributed asynchronous training framework and A-RIDE instead of DigiRL | online training on General tasks · General tasks, 6k seconds | success rate: 30% higher | Helped |
| use DistRL's distributed asynchronous training framework and A-RIDE instead of DigiRL (single) | online training on General tasks · General tasks, 6k seconds | success rate: 40% higher | Helped |
| use DistRL instead of DigiRL-DistRL Async | online training on General tasks · General tasks, 6k seconds | success rate: 10% higher | Helped |
| use DistRL instead of DigiRL (multi) | distributed trajectory collection · training data collection, 6k seconds | trajectories collected: 800 trajectories vs. 300 | Helped |
| use DistRL rather than leading synchronous multi-machine methods | training and data collection · on-device control training | training efficiency: 3× | Helped |
| use DistRL rather than leading synchronous multi-machine methods | training data collection · on-device control training | data collection speed: 2.4× faster | Helped |
| add DPER (ablation) | DistRL · General tasks | success rate: 8% | Helped |
| add Retrace (ablation) | DistRL · General tasks | success rate: 6% | Helped |
Surprising findings
- Adding more trajectory context to the Gemini evaluator reduced evaluation accuracy, as the evaluator tended to hallucinate success from extra screenshots.
Limitations
- Depends on a proprietary Gemini-1.5-Pro evaluator for rewards, which costs budget and may hallucinate success.
- Evaluated only on AitW General and Web Shopping subsets with 128 train and test instructions.
Experimental setup
Datasets: AitW AndroidWorld
Benchmarks: AitW General AitW Web Shopping
Metrics: success rate training efficiency (success vs wall-clock time) trajectory collection rate
Baselines: DigiRL AutoUI AppAgent + GPT-4V AppAgent + Gemini 1.5 Pro DigiRL-DistRL Async
Builds on: DigiRL IMPALA Retrace Distributed Prioritized Experience Replay AutoUI GAE
Closest papers by meaning
- DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents (ICLR 2025 · similarity 0.87)
- WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning (ICLR 2025 · similarity 0.86)
- ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices (CVPR 2026 · similarity 0.86)
- Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control (NeurIPS 2025 · similarity 0.86)
- K²-Agent: Co-Evolving Know-What and Know-How for Hierarchical Mobile Device Control (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.