Research map / Reinforcement learning / Visual and representation RL
Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that IMPALA-style actor-learner training has reproducibility issues from concurrent actor policy updates, and introduces Cleanba, an open-source distributed PPO/IMPALA platform with a synchronized architecture that is faster and more reproducible on Atari.
Problem
Distributed deep RL with actor-learner architectures (e.g., IMPALA) can produce non-reproducible learning curves across hardware settings even with identical hyperparameters, because actor policy versions change mid-rollout.
Approach
Decouple number of environments from CPU count (via EnvPool) and synchronize actor and learner with a queue-based scheme (max queue size 1) so the learner always updates with rollouts from a known policy version (second-latest policy), yielding deterministic rollout composition and reproducible learning curves.
Key results
- Cleanba IMPALA reaches about 165% median HNS on Atari with sticky actions, matching monobeast IMPALA and outperforming moolib IMPALA (140%).
- Under 1 GPU / 10 CPU, Cleanba IMPALA is 6.8x faster than monobeast IMPALA and 1.2x faster than moolib IMPALA.
- Under the workstation setting, Cleanba IMPALA is 5x faster than monobeast IMPALA and 2x faster than moolib IMPALA; Cleanba variants show more consistent learning curves across 1 and 8 GPU settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Cleanba architecture instead of Synchronous architecture (ablation) | Cleanba PPO · Atari | data efficiency | Hurt |
| Use Cleanba architecture instead of Synchronous architecture (ablation) | Cleanba IMPALA · Atari | data efficiency | No clear effect |
| Use Cleanba architecture instead of Synchronous architecture (ablation) | Cleanba IMPALA · Atari | training speed | Helped |
| Use gym.AtariPreprocessing wrappers instead of EnvPool Atari wrappers (ablation) | moolib IMPALA · Atari | median HNS | Hurt |
| Use gym.AtariPreprocessing wrappers instead of EnvPool Atari wrappers (ablation) | moolib IMPALA · Atari | mean HNS | Hurt |
| Use gym.AtariPreprocessing wrappers instead of EnvPool Atari wrappers (ablation) | moolib IMPALA · Atari | IQM HNS | No clear effect |
| Increase moolib IMPALA to 2048 environments, 80 actor CPUs, and 8 GPUs (ablation) | moolib IMPALA · Atari (200M frames) | median HNS: 28.51% median HNS | Hurt |
| Increase the batch size (ablation) | Cleanba · Breakout large-batch training | sample efficiency | Hurt |
| Increase the batch size (ablation) | Cleanba · Breakout large-batch training | time to reach a given score | Helped |
| Use Cleanba IMPALA settings for Cleanba PPO (ablation) | Cleanba PPO · Atari | data efficiency | Hurt |
| Use the same hyperparameters with a 1-second learner delay (ablation) | monobeast IMPALA · Pong | learning-curve reproducibility | Hurt |
| Use Cleanba's synchronized rollout-data composition | distributed IMPALA · Atari, different hardware settings | learning-curve reproducibility | Helped |
Surprising findings
- Cleanba's architecture (parallelized actor-learner) hurts PPO data efficiency, while it does not hurt IMPALA data efficiency.
- Different Atari wrapper implementations have a non-trivial impact on moolib's performance.
Limitations
- Confounding differences in optimizers, learning rates, and frameworks (PyTorch vs JAX) across compared codebases prevent fully controlled comparison.
- The finding that parallel architecture hurts PPO data efficiency is specific to the default Atari hyperparameters.
- Scaling experiments use only one random seed due to hardware constraints.
Experimental setup
Datasets: Atari 57 games
Benchmarks: Atari 57
Metrics: median human normalized score IQM mean HNS optimality gap wall-clock training time learning curve consistency
Baselines: monobeast IMPALA moolib IMPALA CleanRL PPO (Sync) Cleanba PPO (Sync) Cleanba IMPALA (Sync)
Builds on: IMPALA CleanRL EnvPool Podracer Sebulba PPO
Closest papers by meaning
- SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.88)
- Cheaper and Faster: Distributed Deep Reinforcement Learning with Serverless Computing (AAAI 2024 · similarity 0.87)
- Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.85)
- Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependencies (AAAI 2025 · similarity 0.85)
- DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agent (ICLR 2025 · similarity 0.85)
- Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.