Research map / Continual learning / Online and replay-based
Continual Knowledge Adaptation for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CKA-RL, a continual RL method that keeps a pool of task-specific knowledge vectors added to a base policy with softmax-weighted reuse of past vectors, plus cosine-similarity-based merging to bound memory.
Problem
Continual reinforcement learning suffers from catastrophic forgetting, cross-task conflict, inefficient reuse of past knowledge, and memory growth as the number of tasks increases.
Approach
Represent each task's adaptation as a knowledge vector added to a shared base policy; a new task's policy is the base plus a softmax-weighted combination of stored vectors plus a new zero-initialized vector, and the most cosine-similar pair of vectors is averaged whenever the pool exceeds a capacity K_max.
Key results
- CKA-RL achieves average performance 0.7498 vs 0.7196 for CompoNet (about 4.2% relative gain) and average forward transfer 0.5049 vs 0.4674 (about 8%).
- On SpaceInvaders, CKA-RL reaches 0.9928 performance and 0.7749 forward transfer vs 0.9828 and 0.6963 for CompoNet.
- Ablation on Freeway: adding Continual Knowledge Adaptation raises performance from 0.7532 to 0.7821; adding merging gives 0.7923 with a pool of 5 vectors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CompoNet with CKA-RL | continual RL methods · Average | PERF.: 0.7196 → 0.7498 | Helped |
| replace CompoNet with CKA-RL | continual RL methods · Average | FWT.: 0.4674 → 0.5049 | Helped |
| add adaptive weighting of historical knowledge vectors (ablation) | FT-N base on Freeway · Freeway | PERF.: 0.7532 → 0.7821 | Helped |
| add adaptive weighting of historical knowledge vectors (ablation) | FT-N base on Freeway · Freeway | FWT.: 0.6935 → 0.7321 | Helped |
| average knowledge vectors without adaptive weighting (ablation) | FT-N base on Freeway · Freeway | PERF.: 0.7532 → 0.7437 | Hurt |
| average knowledge vectors without adaptive weighting (ablation) | FT-N base on Freeway · Freeway | FWT.: 0.6935 → 0.7127 | Helped |
| add Adaptive Knowledge Merging to Continual Knowledge Adaptation (ablation) | CKA-RL on Freeway · Freeway | PERF.: 0.7821 → 0.7923 | Helped |
| add Adaptive Knowledge Merging to Continual Knowledge Adaptation (ablation) | CKA-RL on Freeway · Freeway | FWT.: 0.7321 → 0.7429 | Helped |
| vary the maximum knowledge-vector pool size from 5 (ablation) | CKA-RL · Freeway and SpaceInvaders | PERF. | Hurt |
| use a larger maximum knowledge-vector pool (ablation) | CKA-RL · Freeway and SpaceInvaders | memory requirements | Hurt |
Surprising findings
- Naively averaging knowledge vectors without adaptive weighting performs worse than the FT-N baseline on Freeway (0.7437 vs 0.7532).
- Forgetting of CKA-RL (0.45) is higher than PackNet (0.04) and MaskNet (0.04), while PackNet has much lower final performance.
Limitations
- Forgetting rate (0.45) is higher than some baselines like PackNet and MaskNet.
- Merging rule is simple averaging of the most similar pair; K_max must be tuned per benchmark (5 for Freeway/SpaceInvaders, 8 for Meta-World).
Experimental setup
Datasets: Meta-World CW20 ALE/SpaceInvaders-v5 ALE/Freeway-v5
Benchmarks: Meta-World SpaceInvaders Freeway
Metrics: average performance forward transfer forgetting
Baselines: FT-1 FT-N ProgNet PackNet MaskNet CReLUs CompoNet CbpNet Baseline
Builds on: CompoNet task arithmetic model editing Continual World
Closest papers by meaning
- Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning (ICLR 2026 · similarity 0.90)
- Knowledge Retention in Continual Model-Based Reinforcement Learning (ICML 2025 · similarity 0.89)
- HTAC: Hierarchical Task-Aware Composition for Continual Offline Reinforcement Learning (ICML 2026 · similarity 0.88)
- From Anti-Forgetting to Fast Adaptation: Continual Reinforcement Learning with World Models (NeurIPS 2026 · similarity 0.88)
- Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.88)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.