Research map / Reinforcement learning / Visual and representation RL
Efficient Reinforcement Learning by Discovering Neural Pathways
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Data Adaptive Pathway Discovery (DAPD), which finds sparse neural pathways (about 5% of parameters) for single-task online SAC and for multi-task online and offline RL, reporting comparable or better returns with much lower FLOPs.
Problem
Large RL networks are costly in energy and compute, and existing sparse training methods for RL are ineffective at high sparsity (95%) and in multi-task settings where tasks share one network.
Approach
Scores weight importance with a SNIP-style gradient saliency computed on the most recent episodic data, periodically re-selects the top-5% mask during a warm-up phase until an episodic return threshold, then freezes it; each task gets its own pathway within one shared network.
Key results
- At 95% sparsity on single-task MuJoCo, SAC-DAPD reaches 9028 on HalfCheetah-v2 versus 8568 for SAC-Dense and 4043 for RigL.
- On MetaWorld MT10 online, SAC-DAPD achieves 77% success versus 49% for SAC-Dense, using 17k parameters versus 340k.
- Offline BCQ+PD reaches 100% success on MT10 with 67k parameters versus 1.34M for the multi-task dense baseline; the paper claims about 20x energy savings from FLOP reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the dense SAC network with a 95%-sparse DAPD pathway | SAC · HalfCheetah-v2 | episodic return: 8568.1 ± 1043.56 -> 9028.02 ± 276.31 | Helped |
| Replace the dense SAC network with a 95%-sparse DAPD pathway | SAC · Walker2d-v2 | episodic return: 2972.49 ± 1691.47 -> 3846.3 ± 459.82 | Helped |
| Replace the dense SAC network with a 95%-sparse DAPD pathway | SAC · Hopper-v2 | episodic return: 3228.5 ± 301.88 -> 3359.88 ±46.57 | Helped |
| Replace the dense SAC network with a 95%-sparse DAPD pathway | SAC · Ant-v2 | episodic return: 3538.22 ± 654.76 -> 3916.65 ± 502.82 | Helped |
| Replace dense multi-task training with separate task-specific sparse pathways | SAC · MetaWorld MT10 | success rate: 49.0 ± 7.3 -> 77 ± 1.3 | Helped |
| Use offline pathway discovery instead of dense multi-task training | BCQ · MetaWorld MT10 | success rate: 81.5 ± 24.15 -> 100 ± 0.0 | Helped |
| Use offline pathway discovery instead of dense multi-task training | IQL · MetaWorld MT10 | success rate: 79.1 ± 26.81 -> 97.3 ± 7.17 | Helped |
| Add a warm-up phase that periodically reconfigures the pathway before training it (ablation) | SAC-DAPD · HalfCheetah-v2 | episodic return | Helped |
| Freeze the pathway after warm-up instead of continuing to update its mask (ablation) | SAC-DAPD · HalfCheetah-v2 | episodic return variance | Helped |
| Continue updating the pathway mask throughout training instead of freezing it after warm-up (ablation) | SAC-DAPD · HalfCheetah-v2 | episodic return variance | Hurt |
Surprising findings
- Dynamic sparse training baselines RigL and RLx2 fail at 95% sparsity in continuous control, performing far below the dense SAC.
- Pathways for different tasks chosen independently still show considerable overlap.
Limitations
- Energy savings are estimated from FLOP counts rather than measured, and rely on hypothetical sparse matrix multiplication support.
- Warm-up threshold and moving-average hyperparameters require tuning; RigL and RLx2 are sensitive to sparsity level.
- Multi-task experiments are limited to small task suites (MT10, five HalfCheetah tasks, four Quadrupod tasks).
Experimental setup
Datasets: MuJoCo HalfCheetah-v2 Walker2d-v2 Hopper-v2 Ant-v2 MetaWorld MT10 Multitask HalfCheetah Multitask Quadrupod
Benchmarks: MuJoCo HalfCheetah-v2 Walker2d-v2 Hopper-v2 Ant-v2 MetaWorld MT10
Metrics: average episodic return success rate normalized score FLOPs estimated energy (Joules)
Baselines: SAC-Dense RigL RLx2 SNIP PCGrad Soft-modularization CARE BCQ-MT BCQ-MHMT CDS
Builds on: SNIP SAC RigL BCQ IQL
Closest papers by meaning
- Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn (NeurIPS 2024 · similarity 0.88)
- On the Role of Computation in Reinforcement Learning (ICML 2026 · similarity 0.88)
- The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.88)
- An Investigation of Model-Free Planning (ICML 2019 · similarity 0.88)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.88)
- Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.