Research map / Continual learning / Online and replay-based
Understanding the Forgetting of (Replay-based) Continual Learning via Feature Learning: Angle Matters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a feature-learning theory of forgetting in continual learning for two-layer polynomial-ReLU CNNs, showing the angle between task signal vectors determines benign vs harmful forgetting, and proposes mid-angle sampling for replay.
Problem
Theoretical understanding of catastrophic forgetting in continual learning is limited; existing analyses mostly use linear models or lazy/NTK regimes, not practical feature-learning neural networks.
Approach
Analyzes sequential binary tasks with a signal-noise data model and a two-layer polynomial ReLU CNN trained by gradient descent, showing forgetting depends on the cosine angle between task signal vectors, and that replay widens the range of angles giving benign forgetting; motivates mid-angle sampling for replay buffers.
Key results
- Synthetic experiments show acute or small obtuse angles give benign forgetting, larger obtuse angles give harmful forgetting, and replay expands the benign range.
- On CIFAR100-10 with iCaRL, mid-angle sampling reaches 48.02% accuracy on previous classes versus 47.17% for random sampling and 47.40% for herding.
- On CIFAR100-10, mid-angle sampling has 14.84% average forgetting versus 15.72% for random sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mid-angle sampling instead of random sampling | iCaRL · CIFAR100-10 | accuracy: 47.17 ± 0.45 -> 48.02 ± 0.27 | Helped |
| Use mid-angle sampling instead of random sampling | iCaRL · CIFAR100-5 | accuracy: 56.08 ± 0.12 -> 56.51 ± 0.06 | Helped |
| Use mid-angle sampling instead of random sampling | iCaRL · CIFAR100-10 | average forgetting: 15.72 ± 0.31 -> 14.84 ± 0.26 | Helped |
| Use mid-angle sampling instead of random sampling | iCaRL · CIFAR100-5 | average forgetting: 11.15 ± 0.46 -> 10.15 ± 0.28 | Helped |
| Add replay to continual learning | Two-layer ReLUq CNN · MNIST sequential task pairs with large obtuse signal angles | test accuracy on Task T1 | Helped |
Surprising findings
- Small obtuse angles yield benign forgetting even without replay, and large obtuse angles cause harmful forgetting unless the replay buffer is large.
Limitations
- Theory restricted to binary classification under a single-head setting.
- Mid-angle sampling contribution is relatively modest, with small gains.
- Analysis assumes signal-noise data model and specific CNN/polynomial ReLU architecture with strong over-parameterization conditions.
Experimental setup
Datasets: MNIST CIFAR100
Benchmarks: MNIST CIFAR100
Metrics: test error on previous task accuracy on previously trained classes average forgetting
Baselines: random sampling herding small-angle sampling big-angle sampling
Builds on: feature learning theory (Allen-Zhu & Li, 2020) Cao et al. 2022 signal-noise data model iCaRL
Closest papers by meaning
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.91)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.90)
- Do Your Best and Get Enough Rest for Continual Learning (CVPR 2025 · similarity 0.90)
- Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks (ICML 2025 · similarity 0.90)
- Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective (ICML 2025 · similarity 0.89)
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.