Research map / Reinforcement learning / Visual and representation RL
Mixtures of Experts Unlock Parameter Scaling for Deep RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces the penultimate layer of value-based deep RL agents (DQN, Rainbow, and others) with Soft MoE or Top-1 MoE modules to make performance scale with parameter count on Atari.
Problem
Scaling parameter count in value-based deep RL often hurts performance, unlike supervised learning where larger models improve; deep RL networks under-utilize their parameters.
Approach
Insert Mixture-of-Experts modules (mainly Soft MoE with per-convolution tokens) in place of the penultimate dense layer of DQN/Rainbow-style networks so that performance improves with the number of experts, while naive widening of the layer degrades performance.
Key results
- In Rainbow, increasing Soft MoE experts from 1 to 8 yields roughly a 20% performance improvement, whereas widening the baseline penultimate layer from 1x to 8x decreases performance by around 40%.
- Soft MoE with CQL+C51 offline attains the best aggregate final performance among the offline variants tested.
- Soft MoE shows no real difference from baseline at 100k interactions but gains when trained for 50M steps, particularly with DER.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the penultimate layer with Soft MoE and increase the expert count from 1 to 8 | Rainbow · 20 Atari games | IQM performance: 20% | Helped |
| Widen the penultimate layer from multiplier 1 to 8 | Rainbow baseline · 20 Atari games | IQM performance: around 40% | Hurt |
| Use Soft MoE instead of the baseline | DQN and Rainbow · 20 Atari games | IQM performance | Helped |
| Use Top1-MoE instead of the baseline | DQN and Rainbow · 20 Atari games | IQM performance | Mixed |
| Increase the number of experts (ablation) | Soft MoE · 20 Atari games | IQM performance | Helped |
| Increase the number of experts (ablation) | Top1-MoE · 20 Atari games | IQM performance | No clear effect |
| Use smaller experts while retaining Soft MoE (ablation) | Soft MoE · DQN and Rainbow, 20 Atari games | IQM performance | No clear effect |
| Replace the learned Soft MoE Φ with a random matrix (ablation) | Soft MoE with one expert · Rainbow | Performance | Hurt |
| Use PerFeat tokenization instead of PerConv (ablation) | Soft MoE · Rainbow, 8 experts | Performance | Hurt |
| Use PerFeat tokenization instead of PerConv (ablation) | Top1-MoE · Rainbow, 8 experts | Performance | Helped |
| Use PerSamp tokenization (ablation) | Soft MoE and Top1-MoE · Rainbow, 8 experts | Performance | Hurt |
| Add load-balancing losses (ablation) | Top1-MoE · DQN and Rainbow | Performance | No clear effect |
Surprising findings
- Widening the baseline penultimate layer hurts performance substantially, while the same capacity added as MoE experts helps.
- Top1-MoE helps only in Rainbow and does not improve with more experts; adding load-balancing losses does not rescue it.
Limitations
- Evaluation focuses on value-based agents on Atari; PPO/SAC experiments in MuJoCo showed no significant gains for Soft MoE.
- Causal link between MoE stabilizing metrics (NTK rank, dormant neurons) and performance is not claimed.
- Top1-MoE does not show parameter scalability, and the work covers single-task off-policy settings only.
Experimental setup
Datasets: Arcade Learning Environment 20-game subset Arcade Learning Environment 60-game set Arcade Learning Environment 26-game Atari 100k set offline DQN replay data (5%, 10%, 50%)
Benchmarks: Atari 20-game subset Atari 60 games Atari 100k (26 games) offline Atari (17 games)
Metrics: interquartile mean (IQM) human-normalized score 95% stratified bootstrap confidence intervals dormant neuron fraction feature rank NTK rank feature norm
Baselines: DQN Rainbow Top1-MoE CQL CQL+C51 DrQ(epsilon) DER widened penultimate layer baseline
Builds on: Soft MoE DQN Rainbow Dopamine Impala ResNet
Closest papers by meaning
- How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization (NeurIPS 2026 · similarity 0.90)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.89)
- Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer (ICML 2025 · similarity 0.89)
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.88)
- Mixture-of-Experts Meets In-Context Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.