Research map / Reinforcement learning / Visual and representation RL
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DigiRL, an autonomous offline-to-online RL approach that fine-tunes a 1.3B VLM (AutoUI-Base) for Android device control using an automatic instruction-level curriculum, doubly-robust step-level advantages, and a VLM-based autonomous evaluator in a parallel Android emulator.
Problem
Off-the-shelf VLMs and static-demonstration imitation learning fail to handle the stochasticity and non-stationarity of real in-the-wild Android/web GUIs for device-control agents.
Approach
Advantage-weighted regression with hard filtering, using a doubly-robust per-step advantage estimator (mixing MC return and learned step value), a cross-entropy-trained value function, and an instruction-level value function that prioritizes tasks of intermediate difficulty, all run on a scalable parallel Android emulator with a Gemini 1.5 Pro autonomous evaluator.
Key results
- DigiRL reaches 67.2% success on AitW Web Shopping test (offline-to-online), versus 17.7% for the best prompting baseline (AppAgent + GPT-4V/Gemini) and 57.8% for Filtered BC (per the abstract/table).
- On AitW General, the offline-to-online DigiRL reaches 71.9% test success versus 61.5% for Filtered BC and 25.0% for CogAgent.
- Parallel emulation scales near-linearly to 64 emulators on 128 CPUs (1.74 vs 0.74 trajs/min naive baseline using 64 CPUs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DigiRL off-to-on RL instead of Filtered BC off-to-on | AutoUI-Base · AitW General Test | success rate: 61.5 -> 71.9 | Helped |
| Use DigiRL off-to-on RL instead of Filtered BC off-to-on | AutoUI-Base · AitW Web Shopping Test | success rate: 57.8 -> 67.2 | Helped |
| Replace offline training with off-to-on RL | DigiRL · AitW General Test | success rate: 62.8 -> 71.9 | Helped |
| Replace offline training with off-to-on RL | DigiRL · AitW Web Shopping Test | success rate: 45.8 -> 67.2 | Helped |
| Use cross-entropy rather than regression for value-function training (ablation) | DigiRL · AitW device-control tasks | performance: around 12% | Helped |
| Add step-level advantages (ablation) | DigiRL · AitW device-control tasks | efficiency: 12% | Helped |
| Add the automatic curriculum (ablation) | DigiRL without step-level advantage · AitW device-control tasks | speed of learning: around 25% | Helped |
| Use hard-filtering instead of vanilla AWR exponential reweighting (ablation) | DigiRL · AitW device-control tasks | performance | No clear effect |
| Add the doubly-robust advantage estimator and automatic curriculum (ablation) | vanilla AWR · AitW device-control tasks | performance | Helped |
| Use distributed emulation across servers rather than running all emulations on one server | Android emulator environment · 64 parallel emulators | trajectories per minute: 0.74 compared to 1.74 trajs/min | Helped |
Surprising findings
- Off-the-shelf GPT-4V and Gemini 1.5 Pro with Set-of-Marks achieve under 20% success, and AppAgent adds at most about 5 points.
- A 1.3B AutoUI-based agent trained with RL outperforms the 18B CogAgent and proprietary VLM wrappers.
Limitations
- Experiments are limited to a subset of AitW tasks (General and Web Shopping), with only 96 test tasks per subset due to budget constraints.
- Results depend on a proprietary Gemini 1.5 Pro autonomous evaluator, and offline data becomes stale quickly as websites change.
Experimental setup
Datasets: Android-in-the-Wild (General) Android-in-the-Wild (Web Shopping)
Benchmarks: Android-in-the-Wild General Android-in-the-Wild Web Shopping
Metrics: task success rate
Baselines: GPT-4V Set-of-Marks Gemini 1.5 Pro Set-of-Marks AppAgent CogAgent AutoUI Filtered BC
Builds on: advantage-weighted regression filtered behavior cloning generalized advantage estimation AutoUI Set-of-Marks
Closest papers by meaning
- Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents (ICLR 2025 · similarity 0.89)
- DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agent (ICLR 2025 · similarity 0.89)
- GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (ICLR 2026 · similarity 0.88)
- What Can RL Bring to VLA Generalization? An Empirical Study (NeurIPS 2025 · similarity 0.87)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.86)
- Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.