Research map / Continual learning / Online and replay-based
Continual Reinforcement Learning by Planning with Online World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Online Agent (OA), which learns a unified world model online via sparse follow-the-leader (Losse-style) linear features with closed-form ridge updates, plans with CEM+MPC, and introduces the Continual Bench spatial multi-task environment for CRL evaluation.
Problem
Continual reinforcement learning requires agents to keep solving earlier tasks while learning new ones, and prior model-based CRL methods suffer catastrophic forgetting and rely on task-indexed dynamics or task IDs; existing CRL benchmarks also have physical conflicts or lack transfer.
Approach
Learn a single unified world-dynamics model online with sparse random-feature (Losse-style) encoding and incremental follow-the-leader ridge updates, which is forgetting-free by construction and has a regret bound, then plan with CEM and MPC for any reward function.
Key results
- OA achieves average performance 72.93 with regret 27.62 on Continual Bench, versus 73.09 AP and 30.95 regret for model-based Perfect Memory and 61.83 AP for Coreset.
- Model-based OA attains lower regret than all model-free and model-based baselines, including Perfect Memory.
- Theory gives a no-regret bound of O(sqrt(K^2 D log T)) for sparse online model learning under stated assumptions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an online FTL world model instead of deep model-based fine-tuning | Model-based planning agent · Continual Bench | AP: 24.86 -> 72.93 | Helped |
| Use an online FTL world model instead of deep model-based fine-tuning | Model-based planning agent · Continual Bench | Reg: 37.74 -> 27.62 | Helped |
| Use online FTL modeling instead of deep model-based perfect memory | Model-based planning agent · Continual Bench | AP: 73.09 -> 72.93 | Mixed |
| Use online FTL modeling instead of deep model-based perfect memory | Model-based planning agent · Continual Bench | Reg: 30.95 -> 27.62 | Helped |
| Add SI regularization instead of deep model-based fine-tuning | Deep model-based planning agent · Continual Bench | AP: 24.86 -> 39.96 | Helped |
| Add coreset replay instead of deep model-based fine-tuning | Deep model-based planning agent · Continual Bench | AP: 24.86 -> 61.83 | Helped |
| Use model-based planning rather than SAC fine-tuning | Fine-tuning agent · Continual Bench | Reg: 67.68 -> 37.74 | Helped |
| Increase model sparsity by using a larger Λ (ablation) | OA · Continual Bench | performance | Helped |
Surprising findings
- Perfect Memory with deep world models matches OA on average performance but has higher regret than OA.
- Existing Continual-World benchmark has physical conflicts between consecutive tasks, which the authors argue undermines studying transfer.
Limitations
- Online world model handles only moderate-dimensional state-based observations and does not model uncertainty.
- Planning does not include explicit exploration.
- Continual Bench is limited to episodic settings with explicit task switches, not reset-free.
Experimental setup
Datasets: Continual Bench
Benchmarks: Continual Bench Continual-World
Metrics: average performance regret world model MSE
Baselines: EWC Synaptic Intelligence PackNet Coreset Perfect Memory Fine-tuning
Builds on: Losse Meta-World Continual-World cross-entropy method MBRL-Lib
Closest papers by meaning
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.89)
- Knowledge Retention in Continual Model-Based Reinforcement Learning (ICML 2025 · similarity 0.89)
- Continual Knowledge Adaptation for Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- The World Is Bigger: A Computationally-Embedded Perspective on the Big World Hypothesis (NeurIPS 2025 · similarity 0.88)
- Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments (NeurIPS 2026 · similarity 0.87)
- PWM: Policy Learning with Multi-Task World Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.