Research map / Continual learning / Online and replay-based
Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes catastrophic forgetting in a two-layer patch-wise CNN trained with gradient descent on sequential binary tasks, using a multi-view data model, and validates with simulations and CIFAR/Tiny-ImageNet experiments.
Problem
Catastrophic forgetting in continual learning is widely observed in CNNs, but its theoretical understanding, especially learning dynamics in non-linear CNNs, remains limited.
Approach
Uses a multi-view data model with task-specific, general, and random features to derive learning dynamics across two sequential tasks in a two-layer CNN, showing forgetting occurs when the task-specific feature is stronger than the general feature and the old task-specific feature appears as a strong random feature in the new task; also gives a theoretical account of replay suppressing forgetting.
Key results
- Under the multi-view model, if the task-specific feature signal exceeds the general feature (αv ≤ o(αu)) and the random feature signal is large (αζ ≥ Ω(αu)), the second-stage model drops to near-random (≤1/2 + o(1)) accuracy on the first task (Theorem 5.4, Corollary 5.5).
- Simulations show that increasing the general feature norm αv suppresses forgetting even when αζ is large, and adding replay data from the first task suppresses forgetting.
- On CIFAR-10, CIFAR-100 and Tiny-ImageNet binary task pairs, the model mostly extracts task-specific rather than shared features, consistent with the theoretical condition αv ≤ o(αu).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a task-specific feature with a stronger signal than the general feature | Two-layer CNN trained on Task 1 · Task 1 | feature learning | Mixed |
| Have the first task's task-specific feature appear as a strong random feature in the second task | Two-layer CNN trained sequentially on two tasks · Task 1 after training on Task 2 | accuracy: high accuracy -> 1 / 2 + O ( nP 2C / poly(d) + 1 / poly(n) ) | Hurt |
| Increase the signal of the shared general feature relative to the task-specific feature | Two-layer CNN trained sequentially on two tasks · Two-task continual learning | accuracy | Helped |
| Replay data points from the previous task during second-stage training | Two-layer CNN trained on two tasks · Task 1 after training on Task 2 | accuracy | Helped |
Surprising findings
- Gradient descent tends to learn features with the strongest signal rather than the most robust ones, so robust general features with weak signal are not captured.
Limitations
- Analysis restricted to two tasks, binary classification, a two-layer CNN with fixed last layer, and task-incremental setting.
- Real-world experiments are qualitative (feature visualization, t-SNE) rather than rigorous tests of the theoretical conditions.
- Assumes over-parameterization, small initialization, and specific SNR conditions.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet simulated multi-view data
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: test accuracy on each task feature inner products with signal directions t-SNE visualization
Builds on: multi-view data model (Allen-Zhu & Li, 2023) replay-based continual learning
Closest papers by meaning
- A comprehensive, application-oriented study of catastrophic forgetting in DNNs (ICLR 2019 · similarity 0.93)
- Understanding the Forgetting of (Replay-based) Continual Learning via Feature Learning: Angle Matters (ICML 2025 · similarity 0.90)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.89)
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.89)
- Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning (ICLR 2025 · similarity 0.89)
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.