Research map / Continual learning / Online and replay-based
Self-Normalized Resets for Plasticity in Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Normalized Resets (SNR), which reset a neuron when its inactivity time is improbable under a geometric model of its own inter-firing times, and supports it with a single-ReLU theory, a hypothesis-test derivation, and continual learning experiments.
Problem
Plasticity loss in continual learning: neural networks trained on a sequence of changing tasks lose the ability to adapt, largely due to inactive (dead) neurons; existing reset and regularization fixes are sensitive to hyperparameters.
Approach
Reset a neuron when the time since its last firing is improbable under a per-neuron geometric model of its inter-firing times (a percentile threshold), derived as the optimal hypothesis test for detecting a dead neuron, instead of using a fixed or layer-relative utility threshold.
Key results
- On SGD, SNR attains 0.851 accuracy on Permuted MNIST vs 0.844 for CBP and 0.710 for no intervention, and 0.975 on Random Label MNIST vs 0.951 for CBP.
- On Permuted Shakespeare over the last 50 tasks, L2 regularization reaches 0.3101 loss while SNR+L2 reaches 0.2551 (lower is better); plain SNR gives 3.0242, roughly no intervention (3.0147).
- At scale 16 of Permuted Shakespeare, L2 has 32.9% dead neurons versus about 1e-6% for SNR+L2, and the SNR+L2 loss is 0.332 vs 0.410 for L2 over the last 50 tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SNR instead of CBP | MLP with SGD · Permuted MNIST | accuracy: 0.844 -> 0.851 | Helped |
| Use SNR instead of CBP | MLP with SGD · Random Label MNIST | accuracy: 0.951 -> 0.975 | Helped |
| Use SNR instead of ReDO | CNN with SGD · Random Label CIFAR | accuracy: 0.981 -> 0.987 | Helped |
| Use SNR instead of ReDO | CNN with SGD · Continual ImageNet | accuracy: 0.869 -> 0.888 | Helped |
| Use SNR instead of L2 Init | MLP with Adam · Permuted MNIST | accuracy: 0.883 -> 0.889 | Helped |
| Use SNR instead of L2 Init | CNN with Adam · Random Label CIFAR | accuracy: 0.976 vs 0.976 | No clear effect |
| Add SNR to L2 regularization | Transformer on Permuted Shakespeare · last 50 tasks | loss: 0.3101 -> 0.2551 | Helped |
| Add SNR to attention-only L2 regularization (ablation) | Transformer on Permuted Shakespeare · all tasks | loss: 0.1506 -> 0.1402 | Helped |
| Add SNR to attention-only L2 regularization (ablation) | ViT on Continual ImageNet · final 50 tasks | training accuracy: 0.91955 -> 0.93372 | Helped |
| Add SNR to L2 regularization | Scaled Transformer on Permuted Shakespeare · scale 16, all tasks | loss: 0.377 -> 0.324 | Helped |
| Add SNR to L2 regularization | Scaled Transformer on Permuted Shakespeare · scale 256, all 50 tasks | training loss: 0.4681 -> 0.4576 | Helped |
| Apply L2 regularization only to attention weights instead of all weights, with SNR (ablation) | Transformer on Permuted Shakespeare · all tasks | loss: 0.2177 -> 0.1402 | Helped |
Surprising findings
- Resets alone are insufficient on Permuted Shakespeare and CI-ViT, where attention layers lack neurons; combining SNR with L2 regularization on attention weights was needed.
- L2 Init performs worse than plain L2 regularization on Permuted Shakespeare, with substantial plasticity loss.
Limitations
- Largest models are about 5M parameters, which is small in a practical sense.
- Theoretical characterization of plasticity loss is limited; the single-ReLU analysis is a first step.
- Resets alone do not prevent plasticity loss in attention-based models, requiring additional L2 regularization.
Experimental setup
Datasets: MNIST CIFAR-10 ImageNet-32 Shakespeare's The Tempest Complete works of Shakespeare (scaled experiment)
Benchmarks: Permuted MNIST Random Label MNIST Random Label CIFAR Continual ImageNet Permuted Shakespeare CI-ViT
Metrics: average accuracy on last 10% of tasks average loss on final epoch of each task training accuracy test accuracy dead neuron rate
Baselines: Continual Backprop (CBP) ReDO L2 regularization L2 Init Shrink and Perturb Layer Normalization
Builds on: Continual Backprop ReDO Shrink and Perturb L2 Init
Closest papers by meaning
- When to Learn and When to Stop: Quitting at the Optimal Time (Student Abstract) (AAAI 2025 · similarity 0.87)
- Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective (ICML 2025 · similarity 0.87)
- Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning (ICML 2025 · similarity 0.87)
- Learning Continually by Spectral Regularization (ICLR 2025 · similarity 0.87)
- Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (ICLR 2024 · similarity 0.87)
- PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.