Research map / Continual learning / Online and replay-based
The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting — An Analytical Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives an exact analytical expression for worst-case expected forgetting in two-task continual linear regression with random orthogonal task transformations, showing task similarity and overparameterization jointly determine forgetting.
Problem
Existing analyses of catastrophic forgetting treat task similarity or overparameterization separately; it is unclear how they jointly affect forgetting, even in elementary models.
Approach
Models the second task as a random orthogonal transformation (over a random m-dimensional subspace) of the first task's data, with DOTS = m/p as task dissimilarity proxy and 1-d/p as overparameterization proxy; derives exact worst-case expected forgetting for SGD-trained two-task linear regression, finding non-monotonic similarity effects only in highly overparameterized regimes.
Key results
- In the highly overparameterized limit, worst-case normalized forgetting equals alpha^2(1-alpha)^2, peaking at intermediate similarity alpha=0.5.
- At the interpolation threshold (d=p), normalized worst-case forgetting equals 2m/p, decreasing monotonically as tasks become more similar.
- On permuted MNIST with a 2-layer MLP of width 400, forgetting peaks at permutation size around 16-24 (intermediate similarity); with lower width (20 MNIST, 40 EMNIST) the peak is less pronounced.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Vary task similarity in the highly overparameterized regime | Two-task linear regression · Two-task linear regression, highly overparameterized | normalized worst-case expected forgetting: \alpha^{2}\left(1-\alpha\right)^{2} | Mixed |
| Vary task similarity at the interpolation threshold | Two-task linear regression · Two-task linear regression, interpolation threshold | normalized worst-case expected forgetting: 2\alpha | Mixed |
| Increase permutation size across similarity levels | 2-layer MLP of width 400 · Permuted MNIST and EMNIST | test error | Mixed |
| Reduce width from 400 to 20 for MNIST and 40 for EMNIST (ablation) | 2-layer MLP trained continually · Permuted MNIST and EMNIST | test error | Mixed |
| Increase task similarity in the average-case linear model with p=3000 | Continual linear regression · Average-case synthetic linear regression, p=3000 | forgetting | Mixed |
| Increase task similarity in the average-case linear model with p=500 | Continual linear regression · Average-case synthetic linear regression, p=500 | forgetting | Mixed |
| Use high overparameterization at task dissimilarity \alpha=0.5 | Two-task linear regression · Two-task linear regression, \alpha=0.5 and \beta\to1 | normalized worst-case expected forgetting: \prn{0.5}^4=0.0625 | Hurt |
Surprising findings
- Overparameterization alone cannot always prevent forgetting; forgetting persists in the highly overparameterized limit depending on task similarity.
- Near the interpolation threshold, forgetting decreases monotonically with task similarity, contradicting prior claims that intermediate similarity is worst.
Limitations
- Analysis restricted to linear regression with two tasks and (S)GD; extension to nonlinear networks and T>=3 tasks is open.
- Overparameterization proxy 1-d/p is specific to linear models and differs from network width.
- Neural network experiments are qualitative and use permuted MNIST/EMNIST only.
Experimental setup
Datasets: MNIST EMNIST permuted MNIST
Benchmarks: permuted MNIST EMNIST
Metrics: forgetting (training loss of first task after second task) test error statistical risk
Baselines: Doan et al. 2021 principal-angle analysis Lee et al. 2021 (permutation-size experiments)
Builds on: Permuted MNIST Evron et al. 2022 continual learning in linear regression (cyclic SGD) Doan et al. 2021 forgetting in linear models Lee et al. 2021 permutation tasks
Closest papers by meaning
- Disentangling and mitigating the impact of task similarity for continual learning (NeurIPS 2024 · similarity 0.91)
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.90)
- Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective (ICML 2025 · similarity 0.90)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.89)
- Understanding the Forgetting of (Replay-based) Continual Learning via Feature Learning: Angle Matters (ICML 2025 · similarity 0.89)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.