Research map / Reinforcement learning / Visual and representation RL
Not All Tasks Are Equally Difficult: Multi-Task Deep Reinforcement Learning with Dynamic Depth Routing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynamic Depth Routing (D2R), a module-level routing framework for multi-task SAC on Meta-World that lets each task use a different number of modules via learned skip connections, with ResRouting and automatic route-balancing.
Problem
Existing multi-task RL routing methods use a fixed number of modules for all tasks, ignoring that tasks of different difficulty need different amounts of knowledge; off-policy training also suffers from mismatched routing paths between behavior and target policies.
Approach
Treat each module as an independent unit in a predetermined topological order, learn a per-task DAG of routing connections (Top-K/SampleK masked softmax) so unused modules are skipped, add ResRouting (stop-gradient with residual shortcut) to handle behavior/target routing mismatch, and use SAC temperatures to set per-task routing temperature for balancing.
Key results
- On MT10-Rand, D2R reaches 0.86 ± 0.05 average success rate versus 0.76 for PaCo and 0.72 for SM, with 1.37M parameters.
- On MT50-Rand, D2R reaches 0.63 ± 0.02 versus 0.54 for PaCo and 0.53 for SM.
- Routing analysis shows difficult tasks use more modules on average (peg-insert-side 10.6 modules vs drawer-close 6.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D2R instead of SM | MTRL · MT10-Fixed | success rate: 0.84 $\pm$ 0.05 -> 0.92 $\pm$ 0.07 | Helped |
| Use D2R instead of PaCo | MTRL · MT10-Rand | success rate: 0.76 $\pm$ 0.04 -> 0.86 $\pm$ 0.05 | Helped |
| Use D2R instead of PaCo | MTRL · MT50-Fixed | success rate: 0.60 $\pm$ 0.03 -> 0.61 $\pm$ 0.03 | Helped |
| Use D2R instead of PaCo | MTRL · MT50-Rand | success rate: 0.54 $\pm$ 0.03 -> 0.63 $\pm$ 0.02 | Helped |
| Use Top-2 routing rather than Hard Routing or Soft Routing (ablation) | D2R · Meta-World | success rate | Helped |
| Remove ResRouting (ablation) | D2R · Meta-World | success rate | Hurt |
| Use newly sampled target-policy routing paths rather than behavior-policy paths (ablation) | D2R · Meta-World | success rate | Hurt |
| Replace SampleK with deterministic TopK during training (ablation) | D2R · Meta-World | success rate | Hurt |
| Remove automatic route-balancing (ablation) | D2R · MT10-Rand | success rate | Hurt |
| Use module-level routing rather than model-level or layer-level routing (ablation) | D2R · Meta-World | success rate | Helped |
| Remove state inputs from the routing network (ablation) | D2R · MT10-Rand | success rate | Hurt |
| Increase module count from below 12 to 12 (ablation) | D2R · MT10-Rand | success rate | Helped |
Surprising findings
- Soft routing (all connections) also converges to sparse paths per input, and Top-2 routing outperforms both Hard (Top-1) and Soft routing.
- Episode length was changed from 150 to 200 steps because 8 Meta-World tasks scored below 90% with rule-based policies at 150 steps.
Limitations
- Number of base modules is predetermined, so some modules may remain unused by any task.
- Few-shot transfer and incremental settings are not evaluated.
Experimental setup
Datasets: Meta-World MT10 Meta-World MT50
Benchmarks: Meta-World MT10-Fixed Meta-World MT10-Rand Meta-World MT50-Fixed Meta-World MT50-Rand
Metrics: average success rate parameter count
Baselines: Multi-task SAC Multi-head SAC SM CARE PaCo
Builds on: Soft Actor-Critic Soft Modularization (SM) PaCo CARE ResNet
Closest papers by meaning
- Efficient Multi-task Reinforcement Learning with Cross-Task Policy Guidance (NeurIPS 2024 · similarity 0.87)
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.86)
- Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning (ICML 2025 · similarity 0.85)
- Dynamic Latent Routing (NeurIPS 2026 · similarity 0.85)
- 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities (NeurIPS 2025 · similarity 0.85)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.