Research map / Continual learning / Online and replay-based
The Importance of Being Lazy: Scaling Limits of Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how width scaling and feature learning (controlled by muP parameter gamma_0) affect catastrophic forgetting in continual learning, showing forgetting depends on the lazy-rich regime and an optimal intermediate feature learning level transfers across widths.
Problem
Existing empirical studies disagree on whether scaling network width reduces catastrophic forgetting in continual learning, and theory of CF has been limited to the lazy regime with fixed features.
Approach
Interpolate between lazy (NTP) and rich (muP) training via a feature-learning factor gamma_0, show width helps forgetting only when feature learning is suppressed, and extend dynamical mean field theory to multi-task sequential training to characterize forgetting in the rich regime.
Key results
- In the lazy NTP parameterization wider networks reduce forgetting, whereas under muP width does not reduce forgetting.
- Optimal plasticity-stability tradeoff occurs at gamma_0* approx 0.1 for both Split-CIFAR10 and Permuted-MNIST, and this value transfers across widths.
- Forgetting shows a sharp low-to-high transition as gamma_0 increases (lazy-rich transition), and higher task similarity shifts this transition and gamma_0* toward 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase width under NTP | ResNet · Split-CIFAR10 | CFr | Helped |
| Increase width under μP | ResNet · Split-CIFAR10 | CFr | No clear effect |
| Increase feature learning (γ₀) beyond the lazy-rich transition | μP models · Split-CIFAR10 | CFr | Hurt |
| Increase feature learning (γ₀) beyond the lazy-rich transition | μP models · Permuted-MNIST | CFr | Hurt |
| Increase feature learning from a low level to an intermediate level | μP models · Split-CIFAR10 | average error: γ₀* ≈ 0.1 | Helped |
| Increase feature learning beyond the optimal intermediate level | μP models · Split-CIFAR10 | average error: γ₀* ≈ 0.1 | Hurt |
| Increase task similarity | μP models · Permuted-MNIST | optimal feature-learning level γ₀* | Helped |
| Increase feature learning for highly similar tasks | ResNet · Split-TinyImagenet | CFr | Helped |
| Increase width under μP at γ₀ below the lazy-rich transition | ResNet · Split-CIFAR10 | CFr | Helped |
| Increase width under μP at γ₀ above the lazy-rich transition | ResNet · Split-CIFAR10 | CFr | No clear effect |
| Use μP+1/√L instead of μP for depth scaling (ablation) | ResNet · Split-CIFAR10 | CFr | Helped |
| Use μP+1/√L instead of μP for depth scaling (ablation) | ResNet · Split-CIFAR10 | average error | Helped |
Surprising findings
- Width scaling reduces forgetting only in the lazy regime; increasing width under rich feature learning does not help and can increase forgetting.
- On highly similar tasks, increasing feature learning beyond a threshold can decrease CFr, a 'pretraining effect'.
Limitations
- Infinite-width theory is validated only on a small 2-layer MLP with 30 samples and 2 tasks due to cubic computational cost.
- Mechanism behind the lazy-rich transition and the inversion of width scaling is hypothesized but not fully explained.
Experimental setup
Datasets: CIFAR10 MNIST TinyImageNet Permuted-MNIST
Benchmarks: Split-CIFAR10 Permuted-MNIST Split-TinyImageNet
Metrics: catastrophic forgetting rate (CFr) learning error average error centered kernel alignment (1-CKA) training loss-based CFr sharpness
Baselines: NTP (standard parameterization) muP
Builds on: Maximal Update Parameterization Neural Tangent Kernel dynamical mean field theory Split-CIFAR10 Centered Kernel Alignment
Closest papers by meaning
- On the Diminishing Returns of Width for Continual Learning (ICML 2024 · similarity 0.90)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.89)
- Understanding the Forgetting of (Replay-based) Continual Learning via Feature Learning: Angle Matters (ICML 2025 · similarity 0.89)
- Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks (ICML 2025 · similarity 0.89)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.89)
- How Feature Learning Can Improve Neural Scaling Laws (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.