Research map / Continual learning / Online and replay-based
Learning Continually by Spectral Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes spectral regularization, which penalizes the largest singular value of each weight matrix toward 1 and biases toward zero, to sustain trainability and mitigate loss of plasticity in continual supervised learning.
Problem
Neural networks lose trainability (plasticity) over the course of continual learning, e.g. on random-label MNIST, as singular value spectra develop outliers and effective rank drops.
Approach
Keep the singular values of network parameters close to their initialization-like near-uniform spectrum by regularizing the largest singular value of each weight matrix toward 1 (and biases toward 0) with a data-independent penalty computed via power iteration, implemented as an optimizer like weight decay.
Key results
- Spectral regularization maintains trainability on random-label MNIST across sequential tasks where standard training loses it, tracked via maximum singular value, minimum singular value and effective rank.
- Spectral regularization is reported to be insensitive to its regularization-strength hyperparameter in sensitivity experiments on random-label MNIST.
- The method is reported to generalize to CIFAR-100 with ResNet-18 and ViT architectures in continual supervised learning.
Limitations
- Trainability is not identical to plasticity/generalization; the authors note the relationship is an open question.
- Evaluated mainly on controlled random-label and standard vision benchmarks; RL application left to future work.
Experimental setup
Datasets: random-label MNIST CIFAR-100 EMNIST
Benchmarks: random-label MNIST CIFAR-100 EMNIST
Metrics: training accuracy maximum singular value minimum singular value effective rank
Baselines: continual backprop L2 regularization towards initialization L2 regularization
Builds on: continual backprop L2 regularization towards initialization spectral normalization
Closest papers by meaning
- Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning (ICML 2026 · similarity 0.91)
- A Statistical Theory of Regularization-Based Continual Learning (ICML 2024 · similarity 0.90)
- Parseval Regularization for Continual Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Optimal Rates in Continual Linear Regression via Increasing Regularization (NeurIPS 2025 · similarity 0.88)
- Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (ICLR 2024 · similarity 0.87)
- Temporal-Difference Variational Continual Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.