Research map / Continual learning / Online and replay-based
Parseval Regularization for Continual Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies Parseval regularization (soft orthogonality of dense-layer weight rows) to PPO/RPO agents to counter plasticity loss in continual RL across gridworld, CARL and Metaworld task sequences.
Problem
Continual RL agents lose trainability (plasticity) when trained on sequences of nonstationary tasks, making learning new tasks progressively harder.
Approach
Adds a soft Parseval penalty ||WW^T - sI||_F^2 to every dense layer except the last, keeping weight matrices near-orthogonal throughout training so that initialization-time gradient-propagation properties persist; paired with diagonal layers or a learnable input scale to restore capacity.
Key results
- On Metaworld20-10 sequences, Parseval regularization substantially outperforms the RPO baseline, Layer Norm, Shrink-and-perturb and Regenerative regularization in performance profiles.
- In gridworld, Parseval regularization lets the agent make progress on almost every task while other methods often stay at 0% success.
- Parseval regularization keeps stable rank near full and neuron cosine similarity near zero, whereas baselines lose rank quickly; runtime overhead is 1.8% to 11.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Parseval regularization | RPO/PPO agents · Gridworld, CARL LunarLander, CARL DMCQuadruped, and Metaworld20-10 | success rate / performance | Helped |
| Use Parseval regularization instead of LayerNorm, Shrink-and-Perturb, or regenerative regularization | Continual RL agents · Gridworld, CARL, and Metaworld | performance | Helped |
| Add diagonal layers to Parseval regularization (ablation) | RPO with Parseval regularization · Metaworld sequences | performance | Helped |
| Add learnable input scaling to Parseval regularization (ablation) | Agent with Parseval regularization · Metaworld sequences | performance | Helped |
| Add diagonal layers or learnable input scaling (ablation) | Base agent without Parseval regularization · Metaworld sequences | performance | Hurt |
| Split neurons into multiple groups for subgroup Parseval regularization (ablation) | Agent with Parseval regularization · Metaworld sequences | performance | Hurt |
| Use only one group rather than multiple groups (ablation) | Agent with Parseval regularization · Metaworld sequences | performance | Helped |
| Regularize only angles between row weight vectors, not their norms (ablation) | Parseval regularization · Metaworld sequences | performance | Helped |
| Add row-norm regularization without angle regularization (ablation) | Baseline agent · Metaworld sequences | performance | Helped |
| Use angle-only or norm-only regularization instead of full Parseval regularization (ablation) | Agent with Parseval regularization · Metaworld sequences | performance | Hurt |
| Increase network width without Parseval regularization (ablation) | Base agent · Metaworld sequences | performance | No clear effect |
| Use Parseval regularization with different activation functions or network widths (ablation) | RPO agent · Metaworld sequences | performance | Helped |
Surprising findings
- Adding either diagonal layers or a learnable input scale alone degrades the base agent, though both help when combined with Parseval regularization.
- Larger initialization scale and policy entropy regularization do not yield the expected gains; the link between weight norms, rank and trainability is found to be complex.
Limitations
- Parseval regularization constrains the network toward orthogonal weights, reducing capacity and Lipschitz flexibility; mitigated only by adding diagonal layers or input scaling.
- Experiments use shallow networks (2 hidden layers, width 64) and simple task-sequence settings with only reward or context changes.
- Subgroup relaxation of the orthogonality constraint did not improve performance.
Experimental setup
Datasets: Metaworld CARL LunarLander CARL DMCQuadruped Gridworld (15x15 navigation)
Benchmarks: Metaworld20-10 CARL LunarLander CARL DMCQuadruped Gridworld
Metrics: success rate performance profiles interquartile mean return stable rank neuron weight cosine similarity runtime
Baselines: Layer Norm Shrink-and-perturb Regenerative regularization W-Regen ReLU CReLU MaxMin Mish
Builds on: orthogonal initialization RPO PPO Shrink-and-perturb Regenerative regularization
Closest papers by meaning
- Learning Continually by Spectral Regularization (ICLR 2025 · similarity 0.90)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.86)
- Regularized Parameter Uncertainty for Improving Generalization in Reinforcement Learning (CVPR 2024 · similarity 0.86)
- A Statistical Theory of Regularization-Based Continual Learning (ICML 2024 · similarity 0.86)
- State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.