Research map / Continual learning / Online and replay-based
Understanding Forgetting in Continual Learning with Linear Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a theoretical upper and matching lower bound on catastrophic forgetting for multi-task SGD in linear regression across under- and overparameterized regimes, and validates with linear and small-DNN simulations.
Problem
Theoretical understanding of catastrophic forgetting in continual learning is limited; prior analyses rely on Gaussian data, noiseless models, or minimum-norm interpolators that require zero training error and overparameterization.
Approach
Analyzes forgetting for multi-step SGD on sequential linear regression tasks under general fourth-moment conditions and well-specified noise, deriving bias-variance upper and lower bounds in terms of covariance spectra, step size, sample size, and task order; shows that training tasks with larger eigenvalues later increases forgetting when data size is large, and that smaller step sizes mitigate f
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train tasks with larger covariance eigenvalues later | linear regression · Synthetic continual learning, under-parameterized | average excess risk (forgetting) | Hurt |
| train tasks with larger covariance eigenvalues later | Deep Neural Networks · Synthetic continual learning | forgetting | Hurt |
| use a smaller step size | linear regression · Synthetic continual learning, across task sequences and dimensionalities | average excess risk (forgetting) | Helped |
| use a smaller step size | Deep Neural Networks · Synthetic continual learning, under-parameterized | forgetting | Helped |
| increase dimensionality | linear regression · Synthetic continual learning, under-parameterized | forgetting | No clear effect |
| increase dimensionality while keeping data size fixed | linear regression · Synthetic continual learning, over-parameterized | forgetting | Hurt |
| increase dimensionality while keeping data size fixed | Deep Neural Networks · Synthetic continual learning, over-parameterized | forgetting | Hurt |
| increase dataset size as dimensionality increases | Deep Neural Networks · Synthetic continual learning, over-parameterized | forgetting | Helped |
Surprising findings
- In over-parameterized DNNs, higher dimensionality does not necessarily increase forgetting at fixed data size, unlike in linear regression.
- Setting the step size to zero does not minimize bias error because the projection term becomes the identity, so a trade-off in step size exists.
Limitations
- Analysis relies on fourth-moment assumptions and well-specified Gaussian noise
- Upper and lower bounds only match when the signal-to-noise ratio is bounded and the step size is small
- Experiments are on synthetic data with only three tasks and a small DNN
Experimental setup
Datasets: synthetic Gaussian-noise linear regression tasks with power-law eigenvalues
Benchmarks: synthetic linear regression tasks synthetic DNN tasks
Metrics: average excess risk (forgetting)
Baselines: minimum-norm interpolator forgetting analysis (Evron et al.) Gaussian-data forgetting analysis (Lin et al.)
Builds on: minimum-norm interpolator forgetting analysis SGD bias-variance analysis in linear regression covariate shift
Closest papers by meaning
- The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting — An Analytical Model (ICLR 2024 · similarity 0.93)
- An Effective Dynamic Gradient Calibration Method for Continual Learning (ICML 2024 · similarity 0.91)
- Optimal Protocols for Continual Learning via Statistical Physics and Control Theory (ICLR 2025 · similarity 0.90)
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.90)
- Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization (ICML 2024 · similarity 0.90)
- Optimal Rates in Continual Linear Regression via Increasing Regularization (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.