Research map / Reinforcement learning / Visual and representation RL
SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SRL, a dataflow abstraction (workers, task handlers, streams, services) and distributed RL system that decouples actor, policy, and trainer workers, achieving up to 21x higher throughput than baselines and scaling to over 15k CPU cores.
Problem
Existing open-source RL systems couple environment simulation, policy inference, and training, leading to low resource efficiency, poor multi-node scaling, and algorithm code entangled with system APIs, which hinders large-scale RL training.
Approach
A general dataflow abstraction (workers hosting task handlers, connected by inference and sample streams and supported by services) where actor, policy, and trainer workers are independently scheduled across heterogeneous nodes, combined with optimizations such as environment rings, trainer pre-fetching, and dynamic batching.
Key results
- SRL reaches up to 21.6x higher maximal distributed training throughput than RLlib (6.3x to 21.6x range).
- SRL reproduces the hide-and-seek solution up to 3x faster than Rapid with CPU inference and up to 5x with GPU inference.
- SRL runs hard hide-and-seek experiments with over 15k CPU cores and 32 A100 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SRL instead of Sample Factory | Single-machine RL training · Atari, DMLab, and SMAC at 64 CPU cores; gFootball at 64 CPU cores favors Sample Factory | training FPS | Mixed |
| Use the decoupled SRL architecture instead of the IMPALA-style architecture | SRL large-scale cluster configuration · Full-sized cluster | training throughput: 3.74x compared to SEED-style and 1.68x compared to IMPALA-style architecture | Mixed |
| Use SRL instead of RLlib | Distributed RL training · Distributed setting | training throughput: 6.3x to 21.6x | Helped |
| Use GPU inference instead of CPU inference (ablation) | SRL actor workers · DMLab and Atari | sample generation throughput: 1.56x and 5.44x speedup ratio | Helped |
| Use GPU inference instead of CPU inference (ablation) | SRL actor workers · gFootball and SMAC | sample generation throughput | No clear effect |
| Add an environment ring rather than using a single environment instance (ablation) | SRL actor workers · Atari actor sample generation | speedup ratio: around 7x | Helped |
| Increase environment ring size (ablation) | SRL actor workers using GPU inference · Actor sample generation | speedup ratio | Helped |
| Enable trainer pre-fetching (ablation) | SRL trainer workers · Atari and DMLab | training throughput: 2-3x | Helped |
| Enable trainer pre-fetching (ablation) | SRL trainer workers · gFootball and SMAC | training throughput | Helped |
| Increase actor-worker count beyond trainer capacity (ablation) | SRL trainer and actor workers · Large-scale sample generation | training FPS | Hurt |
| Scale PPO training with more samples and larger batch sizes | SRL PPO · Atari-5 and gFootball academy scenarios after 8-hour training | IQM performance | Helped |
| Scale Q-learning algorithms with larger batches and more workers under a fixed time budget | SRL Q-learning algorithms · Atari and gFootball benchmarks after 8-hour training | evaluation score | Hurt |
Surprising findings
- Q-learning algorithms (Apex-DQN, VDN) can drop in performance at larger scales under a fixed time budget because they get fewer training steps, while PPO benefits from scale.
Limitations
- Baselines could not be run at the largest scale, so comparisons there re-implement baseline architectures inside SRL rather than using original systems.
- Rapid is closed-source, so hide-and-seek comparisons rely on numbers reported in the original paper.
Experimental setup
Datasets: Atari Google Research Football SMAC DeepMind Lab Hide-and-Seek
Benchmarks: Atari Google Research Football SMAC DeepMind Lab Hide-and-Seek
Metrics: training throughput (FPS) IQM human-normalized score IQM win rate wall-clock speedup to reach ramp lock stage
Baselines: Sample Factory rlpyt SeedRL RLlib MSRL Rapid
Builds on: IMPALA SEED RL PPO Ray RLlib Sample Factory
Closest papers by meaning
- Horizon Reduction Makes RL Scalable (NeurIPS 2025 · similarity 0.90)
- AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (NeurIPS 2025 · similarity 0.90)
- Scaling Offline RL via Efficient and Expressive Shortcut Models (NeurIPS 2025 · similarity 0.89)
- Parallelizing Model-based Reinforcement Learning Over the Sequence Length (NeurIPS 2024 · similarity 0.89)
- AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs (NeurIPS 2026 · similarity 0.89)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.