Research map / Reinforcement learning / Visual and representation RL
Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependencies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TianJi, a distributed DRL training system that relaxes assignment dependencies between actor and learner subtasks via decentralized data-driven training, event-driven asynchronous communication, and a production-consumption balance strategy controlling sample staleness.
Problem
DRL training is slow because assignment dependencies between subtask components (policy inference, simulation, policy update) force serialization and idle time in parallel training systems, while relaxing them introduces convergence uncertainty from stale samples.
Approach
Merge policy inference and simulation into a single actor, convert inter-component dependencies into asynchronous, event-driven data exchanges with clear component boundaries, and control sample staleness by balancing sample production and consumption rates (with resource allocation chosen by performance analysis).
Key results
- TianJi achieves convergence time acceleration ratio up to 4.37 over related systems.
- On eight nodes, TianJi gives 1.6x convergence speedup and 7.13x training throughput speedup over XingTian.
- On on-policy CartPole, TianJi achieves final-time speedups of 4.36 and 2.95 over RLlib and XingTian.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Balance sample production and consumption to control sample staleness (ablation) | TianJi asynchronous PPO training · PPO | convergence time | Helped |
| Use a random resource mapping instead of the strategy-selected mapping (ablation) | TianJi PPO training · PPO | learning performance | Hurt |
| Increase old samples relative to new samples (ablation) | TianJi asynchronous PPO training · PPO | convergence speed | Hurt |
| Increase actors from 4 to 8 to relieve sample-production bottlenecks (ablation) | TianJi with 2 learners · PPO | learning efficiency | Helped |
| Increase actors beyond 8, from L2A8 to L2A16 (ablation) | TianJi with 2 learners · PPO | learning efficiency | No clear effect |
Surprising findings
- Adding more actors beyond a bottleneck (e.g., beyond 16 actors in some settings) does not reduce convergence time, as the learner becomes the bottleneck.
Limitations
- Convergence under relaxed assignment dependencies is not theoretically proven; left as future work.
- Staleness control relies on performance analysis and manual configuration of resource mapping.
- Baseline comparisons limited to a few systems and mostly classic benchmarks.
Experimental setup
Datasets: CartPole Pong Atari games SMAC maps
Benchmarks: CartPole Pong Atari SMAC
Metrics: convergence time throughput episode return acceleration ratio
Baselines: ApeX RLlib XingTian
Builds on: Gorila Ape-X XingTian Prioritized Experience Replay IMPALA-style asynchronous actor-learner
Closest papers by meaning
- SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.88)
- Cheaper and Faster: Distributed Deep Reinforcement Learning with Serverless Computing (AAAI 2024 · similarity 0.86)
- Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform (ICLR 2024 · similarity 0.85)
- Parallelizing Model-based Reinforcement Learning Over the Sequence Length (NeurIPS 2024 · similarity 0.85)
- Compute-Optimal Scaling for Value-Based Deep RL (NeurIPS 2025 · similarity 0.85)
- Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.