Research map / Reinforcement learning / Policy gradient and actor-critic
The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of how parallel environments (N_envs) versus rollout length (N_RO) under fixed or scaled data budgets affect PPO and PQN training on Atari-10, linking the choice to plasticity, feature rank, weight norm and gradient kurtosis.
Problem
How the parallel data-collection configuration (number of parallel environments vs rollout length) and the number of training epochs affect sample efficiency, overfitting, and network plasticity in on-policy deep RL.
Approach
Systematically decouples batch size into parallel environments and rollout length under fixed data budgets and finds that scaling N_envs is more effective than scaling rollout length, with the benefits tied to feature rank, dormant neurons, weight norm, gradient kurtosis, policy variance and ESS.
Key results
- Scaling N_envs (e.g. 8 to 128 at fixed budget) outperforms scaling rollout length N_RO for PPO on Atari-10.
- Doubling N_envs fully offsets the performance decline from doubling epochs (4 to 8) in PPO.
- For PQN, larger N_envs yields no significant return gains, only mild improvements in learning-dynamics metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase N_envs from 8 to 128 while reducing N_RO from 128 to 8 at a fixed data budget (ablation) | PPO · Atari-10 | performance | Helped |
| Increase the data budget by scaling N_envs (ablation) | PPO · Atari-10 | final performance | Helped |
| Increase the data budget by scaling N_RO (ablation) | PPO · Atari-10 | final performance | Helped |
| Increase training epochs from 4 to 8 (ablation) | PPO · Atari-10 | performance | Hurt |
| Double N_envs while training for 8 epochs (ablation) | PPO · Atari-10 | performance | Helped |
| Increase N_envs (ablation) | PPO · Atari-10 | feature rank | Helped |
| Increase N_envs (ablation) | PPO · Atari-10 | dormant neurons | Helped |
| Increase N_envs (ablation) | PPO · Atari-10 | weight norm | Helped |
| Increase N_envs (ablation) | PPO · Atari-10 | gradient kurtosis | Helped |
| Use separate actor and critic encoders rather than shared encoders (ablation) | PPO · Atari-10 | final performance | Mixed |
| Scale learning rate with increased N_envs by k or sqrt(k), rather than using the default learning rate (ablation) | PPO · Atari-10 | performance | No clear effect |
| Vary the discount factor gamma (ablation) | PPO · Atari-10 | performance gains from scaling data collection | No clear effect |
Surprising findings
- For PQN, scaling parallel data gives only marginal performance gains despite mitigating representation deterioration.
- Separate actor-critic encoders alone do not substantially outperform shared encoders unless paired with large-scale data collection.
Limitations
- Fixed N_envs and N_RO throughout training; dynamic scheduling left to future work.
- PQN results show weaker gains than PPO, with mechanism only hypothesized.
- Main analysis focuses on Atari-10 subset of games.
Experimental setup
Datasets: Atari-10 Procgen Isaac Gym
Benchmarks: Atari-10 Procgen Isaac Gym
Metrics: human-normalized IQM feature rank dormant neurons percentage weight norm gradient kurtosis policy variance effective sample size coverage
Baselines: PPO default configuration (N_envs=8, N_RO=128)
Builds on: PPO PQN CleanRL Atari-10
Closest papers by meaning
- Enhancing Diversity In Parallel Agents: A Maximum State Entropy Exploration Story (ICML 2025 · similarity 0.89)
- Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.85)
- A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- Horizon Reduction Makes RL Scalable (NeurIPS 2025 · similarity 0.85)
- Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependencies (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.