Research map / Reinforcement learning / Policy gradient and actor-critic
Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes staggered resets, which initialize and reset parallel environments at varied points within the task horizon so that short-rollout on-policy PPO batches are temporally diverse, improving sample efficiency, wall-clock convergence and scaling in massively parallel RL.
Problem
Synchronous full-episode resets combined with short PPO rollouts (K much smaller than horizon H) in massively parallel GPU simulation make each training batch temporally homogeneous, creating cyclical nonstationarity that destabilizes on-policy learning and limits scaling with more environments.
Approach
Desynchronize parallel environments by advancing each by a staggered offset (discretized into groups spaced by the rollout length) before training and deferring early-termination resets to scheduled reset gates, so every batch covers a diverse range of the task horizon while keeping rollouts short; the method is algorithm-agnostic.
Key results
- On StackCube-v1 with K=8, staggered resets reach >75% success in 16.2M steps versus 35.8M for naive resets (Appendix table).
- Staggered PPO reaches 70% success wall-clock convergence with continued decrease as N grows beyond 6000 environments, whereas naive PPO saturates around N=1024 on StackCube-v1 and degrades at N=6144 on Unitree G1 Transport Box.
- Staggered resets improve SAPG on AllegroKuka tasks in sample efficiency and final reward, and show comparable performance to naive resets on MS-HumanoidWalk.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use staggered rather than synchronous resets | PPO, K=8 · StackCube-v1, 100M environment steps | final reward: 0.62$\pm$0.01 -> 0.74$\pm$0.00 | Helped |
| use staggered rather than synchronous resets | PPO, K=16 · StackCube-v1, 100M environment steps | final reward: 0.72$\pm$0.02 -> 0.72$\pm$0.03 | No clear effect |
| use staggered rather than synchronous resets | PPO, K=32 · StackCube-v1, 100M environment steps | final reward: 0.79$\pm$0.03 -> 0.75$\pm$0.02 | Hurt |
| use staggered rather than synchronous resets | PPO, K=8 · StackCube-v1, >75% success rate | environment steps (millions) to reach threshold: 35.8$\pm$4.5 -> 16.2$\pm$0.2 | Helped |
| use staggered rather than synchronous resets | PPO, K=16 · StackCube-v1, >75% success rate | environment steps (millions) to reach threshold: 18.5$\pm$2.1 -> 17.2$\pm$0.1 | Helped |
| use staggered rather than synchronous resets | PPO, K=8 · StackCube-v1, >75% success rate | wall-clock time (minutes) to reach threshold: 26.2$\pm$2.1 -> 16.6$\pm$3.2 | Helped |
| use staggered rather than synchronous resets | PPO, K=16 · StackCube-v1, >75% success rate | wall-clock time (minutes) to reach threshold: 31.5$\pm$13.6 -> 15.3$\pm$1.0 | Helped |
| increase rollout length from K=8 to K=16 (ablation) | Staggered PPO · StackCube-v1, >75% success rate | wall-clock time (minutes) to reach threshold: 16.6$\pm$3.2 -> 15.3$\pm$1.0 | Helped |
| increase rollout length from K=8 to K=16 (ablation) | Naive PPO · StackCube-v1, >75% success rate | wall-clock time (minutes) to reach threshold: 26.2$\pm$2.1 -> 31.5$\pm$13.6 | Hurt |
| use longer rollouts (K=100) instead of short rollouts (K=8–16) (ablation) | PPO · StackCube-v1 | wall-clock convergence speed | Hurt |
| increase rollout length from K=8 to K=32 (ablation) | Staggered PPO · StackCube-v1, 100M environment steps | final reward: 0.74$\pm$0.00 -> 0.75$\pm$0.02 | No clear effect |
| increase rollout length from K=8 to K=32 (ablation) | Naive PPO · StackCube-v1, 100M environment steps | final reward: 0.62$\pm$0.01 -> 0.79$\pm$0.03 | Helped |
Surprising findings
- Short rollouts (K=8-16) converge 2-3x faster in wall-clock time than long rollouts on StackCube-v1 while matching or exceeding final reward.
- Random reset stochasticity provides only marginal benefit to naive resets in toy environments; deliberate staggering is needed.
Limitations
- Benefit is smaller on locomotion tasks such as MS-HumanoidWalk where natural desynchronization already occurs.
- Staggering requires managing reset schedules and initial advance steps, adding implementation complexity.
- Rollout-length and gating interactions are characterized mainly on toy environments.
Experimental setup
Datasets: ManiSkill3 StackCube-v1 ManiSkill3 PushT ManiSkill3 TwoRobotPushCube ManiSkill3 UnitreeG1 Transport Box ManiSkill3 AnymalC-Reach ManiSkill3 MS-HumanoidWalk IsaacGym AllegroKuka tasks 1D toy chain environment
Benchmarks: ManiSkill3 StackCube-v1 ManiSkill3 PushT ManiSkill3 TwoRobotPushCube ManiSkill3 UnitreeG1 Transport Box ManiSkill3 AnymalC-Reach ManiSkill3 MS-HumanoidWalk IsaacGym AllegroKuka tasks
Metrics: success rate episode reward environment steps to threshold wall-clock time to convergence state visitation KDE
Baselines: Naive synchronous PPO SAPG Long-rollout PPO
Builds on: PPO SAPG ManiSkill3 CleanRL
Closest papers by meaning
- The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks (ICML 2025 · similarity 0.87)
- The Power of Resets in Online Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- SAPG: Split and Aggregate Policy Gradients (ICML 2024 · similarity 0.86)
- Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation (ICLR 2025 · similarity 0.86)
- Enhancing Diversity In Parallel Agents: A Maximum State Entropy Exploration Story (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.