Research map / Continual learning / Online and replay-based
Knowledge Retention in Continual Model-Based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DRAGO, a continual model-based RL method that retains world-model knowledge without stored data via a continually trained VAE generative replay with a frozen old dynamics model, plus an intrinsic-reward "reviewer" policy that revisits states the old model predicts well.
Problem
Continual model-based RL where tasks share dynamics but differ in reward, and agents cannot store past task data, so the world model suffers catastrophic forgetting.
Approach
Combine generative replay (a continually learned VAE producing state-action pairs, labeled next states from a frozen old transition model) with an intrinsic reward for a separate reviewer policy that favors states predicted well by the previous world model but poorly by the current one, to reconnect knowledge across tasks.
Key results
- On few-shot transfer in Cheetah, DRAGO achieves 331.85 vs 73.98 (Continual TDMPC) on jump2back and 106.78 on jump2run, outperforming baselines in 6 of 8 Cheetah/Walker tasks.
- Continual TDMPC performs worse than learning from scratch in overall performance, indicating plasticity loss; EWC improves but remains below DRAGO.
- DRAGO-PETS outperforms continual PETS in 3 of 4 MiniGrid dense-reward transfer tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Synthetic Experience Rehearsal and Regaining Memories Through Exploration to continual TDMPC | TDMPC · Cheetah and Walker few-shot transfer tasks | average reward | Mixed |
| Remove Synthetic Experience Rehearsal (ablation) | DRAGO · Cheetah and MiniGrid transfer tasks | average reward | Hurt |
| Remove Regaining Memories Through Exploration (reviewer) (ablation) | DRAGO · Cheetah and MiniGrid transfer tasks | average reward | Hurt |
| Remove Q-value estimation from planning (ablation) | Continual TDMPC · Continual transfer tasks | performance | Hurt |
| Use one-step rather than multi-step prediction loss for Q and reward updates (ablation) | Continual TDMPC · Cheetah jump | performance | Helped |
| Add the reviewer intrinsic reward directly to the learner's external reward instead of using a separate reviewer (ablation) | DRAGO · Cheetah continual training tasks | average episode return: 652.53 -> 583.13; 587.24 -> 403.70; 624.09 -> 330.73 | Hurt |
| Add a curiosity intrinsic reward to continual TDMPC | Continual TDMPC · Cheetah transfer tasks | average reward: 88.36±25.81 vs 106.78±32.01; 165.35±67.01 vs 248.92±15.38; 133.81±23.07 vs 331.85±11.05; 138.77±45.55 vs 147.30±34.29 | Hurt |
| Use DRAGO-PETS instead of Continual PETS | PETS · MiniGrid transfer tasks | average reward (dense): 233.21±21.07 vs 150.84±62.37; 101.03±116.21 vs 161.70±44.31; 138.26±99.05 vs 43.04±111.93; 234.65±41.71 vs 147.80±105.84 | Mixed |
| Load the prior Cheetah run policy, value, and reward models in addition to the world model (ablation) | DRAGO · Cheetah jump2run | performance | Helped |
| Train DRAGO without Synthetic Experience Rehearsal (ablation) | DRAGO · Cheetah jump&runforward | performance | No clear effect |
Surprising findings
- Continual TDMPC constantly performs worse than learning from scratch, indicating severe plasticity loss in naive continual MBRL.
- Directly adding the reviewer's intrinsic reward to the learner's external reward substantially degrades continual training performance (e.g., Cheetah run 583 vs 653).
Limitations
- Single generative model is continually updated and may suffer mode collapse or blurring as the number of tasks grows.
- Evaluated only on relatively simple tasks and a limited number of tasks.
- Does not fully alleviate plasticity loss; learning from scratch still performs best on some tasks (Cheetah jump&back).
Experimental setup
Datasets: MiniGrid 27x27 four-room gridworld tasks DeepMind Control Suite Cheetah and Walker tasks
Benchmarks: MiniGrid transfer tasks DeepMind Control Cheetah transfer tasks DeepMind Control Walker transfer tasks
Metrics: average return cumulative reward at end of few-shot training prediction score (MSE-based heatmaps)
Baselines: Scratch TDMPC Continual TDMPC EWC Replay-based MBRL Pseudo-rehearsal MBRL Continual TDMPC + Curiosity Continual PETS
Builds on: TD-MPC Elastic Weight Consolidation Variational autoencoder Model Predictive Path Integral
Closest papers by meaning
- Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.89)
- From Anti-Forgetting to Fast Adaptation: Continual Reinforcement Learning with World Models (NeurIPS 2026 · similarity 0.89)
- Continual Knowledge Adaptation for Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Continual World: A Robotic Benchmark For Continual Reinforcement Learning (NeurIPS 2021 · similarity 0.88)
- Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning (ICML 2026 · similarity 0.87)
- Self-Composing Policies for Scalable Continual Reinforcement Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.