Research map / Reinforcement learning / Policy gradient and actor-critic
Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Succeed or Learn Slowly (SoLS), an off-policy actor-critic RL method that applies full updates to positive-advantage samples and PPO-style clipped updates to negative ones, combined with Successful Transition Replay (STR), to fine-tune Llama-3-8B for AndroidWorld mobile app control.
Problem
Applying RL to fine-tune foundation models for multi-turn mobile app control is hard due to sparse rewards, expensive emulator simulation, and a large token-level action space where negative-advantage policy-gradient updates can degrade the model.
Approach
Asymmetric off-policy actor-critic update: positive-advantage transitions are updated directly (no policy regularisation), while negative-advantage transitions get conservative PPO-like clipped updates, paired with a replay buffer (STR) that stores only successful timesteps per task.
Key results
- SoLS-STR achieves 51.3% overall success rate on an 80-task AndroidWorld subset, versus 38.8% for DigiRL-STR and 32.1% for A2C-STR (three runs, ±1.2 SEM).
- Best GPT-4o-based method UGround-V1-7B reaches 43.8% overall success rate; SoLS-STR is at least 17% relatively higher.
- SoLS inference takes about 0.9 seconds per step, roughly 60x faster than UGround-7B's 40-60 seconds per step.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SoLS asymmetric policy updates with STR instead of DigiRL-STR | Llama-3-8B-Instruct · AndroidWorld 80-task subset | overall success rate: 38.8 ± 0.0 -> 51.3 ± 1.2 | Helped |
| Use SoLS asymmetric policy updates with STR instead of A2C-STR | Llama-3-8B-Instruct · AndroidWorld 80-task subset | overall success rate: 32.1% vs 51.3% | Helped |
| Add Successful Transition Replay (STR) (ablation) | PPO, DigiRL, and SoLS · AndroidWorld 80-task subset | success rate | Helped |
| Use asymmetric policy updates instead of standard A2C updates for negative-advantage actions (ablation) | A2C-STR · AndroidWorld 80-task subset | overall success rate: 32.1% vs 51.3% | Helped |
| Use SoLS instead of PPO when trained exclusively on on-policy data (ablation) | AndroidWorld RL training · AndroidWorld 80-task subset | success rate | Helped |
| Add STR to PPO (ablation) | PPO · AndroidWorld 80-task subset | success rate | Helped |
Surprising findings
- SoLS without STR still outperforms PPO and DigiRL trained purely on-policy, and is on par with DigiRL-STR.
- Standard A2C with STR reaches only 32.1%, far below SoLS-STR's 51.3%, suggesting that unrestricted negative-advantage updates are harmful.
Limitations
- Asymmetric constraint may lead to premature convergence to local optima when early positive advantages are noisy.
- Performance depends heavily on the quality of the initial SFT policy and cannot learn interaction patterns absent from the initial policy and exploration distribution.
- Text-only inputs without screenshots or observation history fail on memory-, visual-, and long-horizon tasks.
Experimental setup
Datasets: AndroidControl AndroidWorld
Benchmarks: AndroidWorld
Metrics: success rate inference time
Baselines: SeeAct T3A M3A UGround-V1 AriaUI OS-Atlas-Pro SFT PPO A2C-STR DigiRL-STR
Builds on: PPO DigiRL off-policy actor-critic WoLF A2C
Closest papers by meaning
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.86)
- Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.86)
- DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agent (ICLR 2025 · similarity 0.86)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.86)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.