Research map / Reinforcement learning / Visual and representation RL
Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies data augmentation, network modules, and training stages for plasticity loss in visual RL, and proposes Adaptive RR, which raises the replay ratio once the critic's fraction of active units recovers.
Problem
Plasticity loss in visual reinforcement learning limits sample efficiency, and a high replay ratio (RR) that should improve sample efficiency is offset by worsening plasticity loss, leaving the choice of a static RR unresolved.
Approach
Plasticity loss is diagnosed along three facets: data augmentation preserves plasticity, the critic (not the encoder or actor) is the main bottleneck, and early-stage critic plasticity loss is irrecoverable. Based on this, the RR is adapted online, starting low and switching to a high value once the critic's fraction of active units plateaus.
Key results
- Adaptive RR reaches 880 on Cheetah Run, 758 on Walker Run and 784 on Quadruped Run after 2M steps, versus 511 for static RR=2 on Quadruped Run.
- On Atari-100k over 17 games, Adaptive RR attains 55.8% mean HNS and 48.7% median HNS, against 35.1% to 42.3% mean HNS for static DrQ(epsilon) settings.
- Turning DA off after the critic's plasticity has recovered does not hurt training, while turning DA on late cannot revive performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase RR dynamically from 0.5 to 2 after critic plasticity recovers, rather than using a static RR | DrQ-v2 · Atari-100k, 17 games | Mean HNS (%): 42.3 -> 55.8 | Helped |
| Increase RR dynamically from 0.5 to 2 after critic plasticity recovers, rather than using a static RR | DrQ-v2 · Atari-100k, 17 games | Median HNS (%): 41.6 -> 48.7 | Helped |
| Use data augmentation | DrQ-v2 · DeepMind Control Suite | training performance | Helped |
| Add Reset when data augmentation is absent (ablation) | DrQ-v2 · DeepMind Control Suite | training performance | Helped |
| Add Reset when data augmentation is applied (ablation) | DrQ-v2 · DeepMind Control Suite | training performance | Mixed |
| Use data augmentation instead of no augmentation with a frozen ImageNet pre-trained encoder (ablation) | DrQ-v2 with frozen ResNet encoder · DeepMind Control Suite | sample efficiency | Helped |
| Inject plasticity into the critic after 1M steps without data augmentation (ablation) | DrQ-v2 · Walker Run | training performance | Helped |
| Inject plasticity into the actor or critic when using data augmentation (ablation) | DrQ-v2 · Walker Run | training performance | No clear effect |
| Turn off data augmentation after critic plasticity recovers (ablation) | DrQ-v2 · DeepMind Control Suite | training efficiency | No clear effect |
| Turn on data augmentation only after substantial plasticity loss in the early stage (ablation) | DrQ-v2 · DeepMind Control Suite | training performance | No clear effect |
| Increase static RR from 0.5 to 1 (ablation) | DrQ-v2 · DeepMind Control Suite | sample efficiency | Helped |
| Increase static RR beyond 1 (ablation) | DrQ-v2 · DeepMind Control Suite | sample efficiency | Hurt |
Surprising findings
- Reset improves performance without data augmentation but gives limited or negative gains when data augmentation is used.
- Plasticity injection on the critic yields a large gain without data augmentation, while the same intervention has no effect when data augmentation is applied.
Limitations
- Evaluation is limited to DMC and Atari environments, without more complex settings.
- Adaptive RR is only demonstrated under basic configurations with a single switch from low to high RR.
- FAU may be limited as a plasticity metric for convolutional networks.
Experimental setup
Datasets: DeepMind Control Suite Atari-100k ImageNet
Benchmarks: DeepMind Control Suite Atari-100k
Metrics: average episode return fraction of active units human-normalized score mean and median HNS number of superhuman and best games
Baselines: DrQ-v2 with static RR=0.5 DrQ-v2 with static RR=2 Reset ReDo DrQ(epsilon) RR=0.5/1/2
Builds on: DrQ-v2 ReDo Plasticity Injection L2-Init DrQ(epsilon)
Closest papers by meaning
- A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.88)
- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.88)
- Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn (ICML 2025 · similarity 0.88)
- When Should We Prefer State-to-Visual DAgger over Visual Reinforcement Learning? (AAAI 2025 · similarity 0.87)
- Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.