Research map / Continual learning / Online and replay-based
Self-Composing Policies for Scalable Continual Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CompoNet, a growable modular policy network where each new module attends over and composes frozen outputs of previous policies, achieving linear parameter growth and positive forward transfer in continual RL.
Problem
Continual reinforcement learning must avoid catastrophic forgetting and interference while transferring knowledge, but growing network approaches grow quadratically in parameters with the number of tasks, limiting scalability.
Approach
Each new module consists of an output attention head that linearly combines previous modules' outputs, an input attention head over those outputs, and an internal policy that adjusts the tentative output; modules learn to compose themselves without a separate composition network, giving constant per-module parameters and linear total growth.
Key results
- On Meta-World, CompoNet achieves 0.42 performance and 0.01 forward transfer, the only method with positive forward transfer, versus ProgressiveNet 0.41 and -0.04.
- On SpaceInvaders, CompoNet reaches 0.99 performance and 0.74 forward transfer, above the reference transfer RT of 0.70.
- On Freeway, CompoNet reaches 0.94 performance and 0.80 forward transfer, versus FT-N 0.81 performance and ProgressiveNet 0.47 performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CompoNet instead of ProgressiveNet | Continual reinforcement learning · Meta-World | performance: 0.41 ± 0.49 -> 0.42 ± 0.49 | Helped |
| Use CompoNet instead of ProgressiveNet | Continual reinforcement learning · Meta-World | forward transfer: -0.04 ± 0.04 -> 0.01 ± 0.14 | Helped |
| Use CompoNet instead of FT-N | Continual reinforcement learning · SpaceInvaders | performance: 0.99 ± 0.01 -> 0.99 ± 0.01 | No clear effect |
| Use CompoNet instead of FT-1 | Continual reinforcement learning · SpaceInvaders | forward transfer: 0.73 ± 0.25 -> 0.74 ± 0.22 | Helped |
| Use CompoNet instead of FT-N | Continual reinforcement learning · Freeway | performance: 0.81 ± 0.01 -> 0.94 ± 0.06 | Helped |
| Use CompoNet instead of FT-1 | Continual reinforcement learning · Freeway | forward transfer: 0.64 ± 0.09 -> 0.80 ± 0.07 | Helped |
Surprising findings
- Empirical inference time of CompoNet does not show quadratic growth up to 300 tasks, despite a theoretical O(n^2) inference cost.
- CompoNet matches or slightly improves training from scratch when all previous modules are non-informative random policies.
Limitations
- Theoretical inference cost is quadratic in the number of tasks.
- Requires known task boundaries and task identifiers.
- Image-based tasks use per-module CNN encoders, and the shared visual foundation model option was only preliminarily explored.
Experimental setup
Datasets: Meta-World ALE/SpaceInvaders-v5 ALE/Freeway-v5
Benchmarks: Meta-World CW20-style sequence SpaceInvaders mode sequence Freeway mode sequence
Metrics: average performance forward transfer reference forward transfer success rate
Baselines: FT-1 FT-N ProgressiveNet PackNet Baseline (train from scratch)
Builds on: ProgressiveNet Soft Actor-Critic Proximal Policy Optimization scaled dot-product attention Continual World
Closest papers by meaning
- Policy Consolidation for Continual Reinforcement Learning (ICML 2019 · similarity 0.90)
- Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning (AAAI 2025 · similarity 0.87)
- Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.86)
- Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning (ICML 2025 · similarity 0.86)
- Knowledge Retention in Continual Model-Based Reinforcement Learning (ICML 2025 · similarity 0.86)
- Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.