Research map / Reinforcement learning / Visual and representation RL
On-Policy Algorithms for Continual Reinforcement Learning (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares PPO and SAC with fine-tuning, L2, EWC and PackNet on a 5-task Continual World sequence, finding PPO has lower average performance but competitive forward transfer and forgetting.
Problem
Continual reinforcement learning has focused on off-policy algorithms with replay buffers, which mitigate distribution shift; it is unclear how on-policy algorithms such as PPO behave in continual RL settings.
Approach
Applies the Continual World benchmark setup to PPO (on-policy) and directly compares it with SAC (off-policy) under standard continual learning methods, using the CRL metrics of performance, forward transfer and forgetting.
Key results
- PPO average performance with EWC is 0.375 versus 0.817 for SAC with EWC over five tasks.
- PPO with fine-tuning shows no forward transfer (-0.007) while SAC fine-tuning gets 0.458 transfer.
- PackNet with PPO shows forgetting of 0.073 whereas PackNet with SAC shows -0.011, i.e. no forgetting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PPO instead of SAC with fine-tuning | Continual World continual-learning setup · Continual World, 5-task sequence | Performance: 0.212 -> 0.164 | Hurt |
| Use PPO instead of SAC with fine-tuning | Continual World continual-learning setup · Continual World, 5-task sequence | Transfer: 0.458 -> -0.007 | Hurt |
| Use PPO instead of SAC with fine-tuning | Continual World continual-learning setup · Continual World, 5-task sequence | Forgetting: 0.764 -> 0.271 | Helped |
| Use PPO instead of SAC with L2 regularization | Continual World continual-learning setup · Continual World, 5-task sequence | Performance: 0.639 -> 0.348 | Hurt |
| Use PPO instead of SAC with L2 regularization | Continual World continual-learning setup · Continual World, 5-task sequence | Transfer: -0.886 -> -0.138 | Helped |
| Use PPO instead of SAC with L2 regularization | Continual World continual-learning setup · Continual World, 5-task sequence | Forgetting: 0.072 -> -0.016 | Helped |
| Use PPO instead of SAC with EWC | Continual World continual-learning setup · Continual World, 5-task sequence | Performance: 0.817 -> 0.375 | Hurt |
| Use PPO instead of SAC with EWC | Continual World continual-learning setup · Continual World, 5-task sequence | Transfer: -0.127 -> -0.050 | Helped |
| Use PPO instead of SAC with PackNet | Continual World continual-learning setup · Continual World, 5-task sequence | Performance: 0.842 -> 0.387 | Hurt |
| Use PPO instead of SAC with PackNet | Continual World continual-learning setup · Continual World, 5-task sequence | Transfer: -0.735 -> -0.053 | Helped |
| Use PPO instead of SAC with PackNet | Continual World continual-learning setup · Continual World, 5-task sequence | Forgetting: -0.011 -> 0.073 | Hurt |
| Apply L2 regularization instead of fine-tuning | PPO · Continual World, 5-task sequence | Performance: 0.164 -> 0.348 | Helped |
Surprising findings
- PackNet with SAC shows no forgetting, while PackNet with PPO displays some forgetting.
- With CL methods, PPO shows higher forward transfer than SAC, despite lower average performance.
Limitations
- Small study with a single five-task sequence on one benchmark
- Authors do not identify the precise cause of PPO's behaviour
- Proposed CL methods for on-policy algorithms are left to future work
Experimental setup
Datasets: Continual World
Benchmarks: Continual World
Metrics: average performance forward transfer forgetting
Baselines: SAC fine-tuning L2 regularization EWC PackNet
Builds on: Continual World PPO SAC elastic weight consolidation PackNet
Closest papers by meaning
- Policy Optimization for Continuous Reinforcement Learning (NeurIPS 2023 · similarity 0.89)
- Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Position: Lifetime tuning is incompatible with continual reinforcement learning (ICML 2025 · similarity 0.86)
- REBEL: Reinforcement Learning via Regressing Relative Rewards (NeurIPS 2024 · similarity 0.86)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.