Research map / Reinforcement learning / Visual and representation RL
Cheaper and Faster: Distributed Deep Reinforcement Learning with Serverless Computing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MINIONSRL, a serverless distributed DRL training framework on Azure Container Instances that uses a PPO-trained scheduler to dynamically scale actor functions per round, cutting training time and cost versus server-based baselines.
Problem
On-policy distributed DRL with actor-learner architectures wastes resources because actors sit idle during synchronization and policy updates on coarse-grained server-based clusters, inflating cloud cost.
Approach
Refactor the actor-learner DRL pipeline into serverless functions with fine-grained, pay-per-use billing, and learn a PPO-based scheduler that chooses how many actor functions to launch in each training round to trade off training time against cost.
Key results
- MinionsRL reduces total training time by up to 52% and training cost by up to 86% compared to Azure ML and IMPACT.
- Compared to the static MinionsRL-Max variant, MinionsRL reduces training cost by up to 44% at similar duration.
- Fine-tuning a scheduler from another task reduces scheduler training time and cost by 90% while matching or beating training from scratch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use serverless MINIONSRL rather than Azure ML | PPO distributed training · OpenAI Gym tasks | training time: reduces total training time by up to 52% | Helped |
| use serverless MINIONSRL rather than Azure ML | PPO distributed training · OpenAI Gym tasks | training cost: reduces training cost by up to 86% | Helped |
| use dynamic DRL-based actor scheduling instead of 32 actors every round (ablation) | MINIONSRL · Hopper-v3 | training cost: 1.7 ± 0.3 vs 1.2 ± 0.2 | Helped |
| use dynamic DRL-based actor scheduling instead of 32 actors every round (ablation) | MINIONSRL · QbertNoFrameskip-v4 | training cost: 5.1 ± 1.2 vs 2.3 ± 0.8 | Helped |
| replace reward-ratio scheduling with DRL-guided actor scheduling (ablation) | MINIONSRL · Hopper-v3 | training time: 403 ± 45 vs 241 ± 24 | Helped |
| replace reward-ratio scheduling with DRL-guided actor scheduling (ablation) | MINIONSRL · QbertNoFrameskip-v4 | training time: 750 ± 61 vs 506 ± 59 | Helped |
| fine-tune a scheduler from another task instead of training it from scratch (ablation) | MINIONSRL scheduler · Humanoid-v3 and SpaceInvadersNoFrameskip-v4 | scheduler training time: reducing the scheduler training time and cost by 90% | Helped |
| increase the number of actors every ten rounds instead of using a fixed count (ablation) | PPO on CartPole-v1 · CartPole-v1 | final reward | Mixed |
| use serverless MINIONSRL rather than IMPACT | PPO distributed training · HalfCheetah-v3 | training time: 220 ± 29 vs 193 ± 12 | Hurt |
Limitations
- Scheduler must be trained separately for each DRL task, with high overhead (10-50 hours from scratch).
- Evaluated only on six OpenAI Gym tasks with PPO as the learner optimizer on Azure.
Experimental setup
Datasets: OpenAI Gym Hopper-v3 OpenAI Gym Humanoid-v3 OpenAI Gym HalfCheetah-v3 OpenAI Gym GravitarNoFrameskip-v4 OpenAI Gym SpaceInvadersNoFrameskip-v4 OpenAI Gym QbertNoFrameskip-v4
Benchmarks: Hopper-v3 Humanoid-v3 HalfCheetah-v3 GravitarNoFrameskip-v4 SpaceInvadersNoFrameskip-v4 QbertNoFrameskip-v4
Metrics: total training time (s) training cost ($) final rewards scalability round completion time latency breakdown
Baselines: Azure ML distributed PPO IMPACT MinionsRL-Max MinionsRL-Adapt
Builds on: PPO IMPALA IMPACT Ray RLlib
Closest papers by meaning
- SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.88)
- Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform (ICLR 2024 · similarity 0.87)
- Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependencies (AAAI 2025 · similarity 0.86)
- JaxMARL: Multi-Agent RL Environments and Algorithms in JAX (NeurIPS 2024 · similarity 0.85)
- AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (NeurIPS 2025 · similarity 0.84)
- AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.