Research map / Reinforcement learning / Visual and representation RL
A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Forget-and-Grow (FoG), combining experience replay decay (down-weighting old transitions) with network expansion (adding residual blocks to the critic during training) on top of OBAC to mitigate primacy bias in continuous control.
Problem
Primacy bias in off-policy deep RL for continuous control: agents overfit early replay-buffer experiences under high replay ratios, and periodic resets alone do not fully resolve it.
Approach
Mimics infantile amnesia with two mechanisms: ER decay lowers sampling weight of older transitions (with a minimum weight floor), and network expansion adds new residual critic blocks early in training after each reset, built on an OBAC backbone with scaled critic and replay ratio 10.
Key results
- FoG achieves a total normalized score of 0.92 ± 0.01 across 41 tasks, versus 0.76 for BRO, 0.70 for TD-MPC2 and 0.69 for SimBa.
- On DMC-Hard, FoG reaches 0.99 normalized score versus 0.76 for BRO and 0.75 for SimBa.
- FoG scales to about 23M parameters, and the paper reports that network expansion, not fixed-size models, gives the performance gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ER decay and network expansion to SAC with reset (Expanded SAC) | SAC with reset · humanoid-walk | final score: +53% | Helped |
| Add ER decay and network expansion to SAC with reset (Expanded SAC) | SAC with reset · HalfCheetah-v4 | final score: +27% | Helped |
| Add ER decay and network expansion to SAC with reset (Expanded SAC) | SAC with reset · humanoid-walk | final score vs SimBa: +74% | Helped |
| Add ER decay and network expansion to SAC with reset (Expanded SAC) | SAC with reset · HalfCheetah-v4 | final score vs SimBa: +22% | Helped |
| Replace uniform replay with ER decay (ablation) | FoG · Mujoco and DMControl tasks | final performance | Helped |
| Replace uniform replay with PER (ablation) | FoG · Mujoco and DMControl tasks | final performance | Helped |
| Replace uniform replay with CUER (ablation) | FoG · Mujoco and DMControl tasks | final performance | No clear effect |
| Disable experience replay method (ablation) | FoG · Mujoco and DMControl tasks | final performance | Mixed |
| Disable network expansion and use a fixed model with 2 additional blocks (ablation) | FoG · Mujoco and DMControl tasks | performance | Hurt |
| Disable network expansion and use a fixed model with 4 additional blocks (ablation) | FoG · Mujoco and DMControl tasks | performance | Hurt |
| Remove ER decay and network expansion from SAC with reset (ablation) | Expanded SAC · humanoid-walk and HalfCheetah-v4 | final score: 53%; 27% | Hurt |
| Apply ER decay to SAC (ablation) | SAC · Mujoco tasks | performance | Helped |
Surprising findings
- Resetting alone does not resolve primacy bias: SAC with reset still shows loss spikes on early data, motivating the new mechanisms.
- FoG-BRO performed nearly identically to the original BRO, suggesting some BRO adjustments conflict with the FoG mechanism.
Limitations
- Increased computational cost and longer training time limit use where rapid training is needed.
- Theoretical justification is limited to intuitive arguments and simple bounds; mechanisms are not deeply analyzed.
- FoG-BRO and FoG-SAC gains were limited, so the method may not transfer universally to other algorithms.
Experimental setup
Datasets: MuJoCo DMControl Meta-World HumanoidBench
Benchmarks: MuJoCo DMControl Meta-World HumanoidBench
Metrics: return success rate normalized score critic loss dormant neuron ratio
Baselines: BRO SimBa TD-MPC2 SAC PER CUER Neuroplastic Expansion
Builds on: OBAC SAC BRO plasticity injection prioritized experience replay
Closest papers by meaning
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.88)
- Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning (ICML 2025 · similarity 0.87)
- Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.87)
- The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning (ICML 2025 · similarity 0.87)
- Prioritized Generative Replay (ICLR 2025 · similarity 0.87)
- Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous control (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.