Research map / Continual learning / Online and replay-based
Disentangling and mitigating the impact of task similarity for continual learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes continual learning of partially similar tasks in a linear teacher-student model with latent structure, disentangling input-feature and readout similarity and testing gating and Fisher-metric weight regularization, validated on a latent permuted MNIST.
Problem
How input-feature similarity and readout similarity between sequential tasks affect knowledge transfer and catastrophic forgetting, and how common continual learning algorithms (activity/plasticity gating, weight regularization) interact with that similarity.
Approach
Introduces a low-dimensional latent variable into a linear teacher-student model so that feature correlation and readout correlation can be controlled separately, yielding closed-form transfer and retention expressions; analyzes how gating and Fisher-metric regularization change them.
Key results
- High feature similarity coupled with low readout similarity yields negative transfer and retention (e.g., Delta eps_TF = rho_a(2rho_b - rho_a) becomes negative when rho_a > 2rho_b).
- Fisher-metric weight regularization keeps retention near perfect regardless of task similarity when N_s << N_x(1-rho_a^2), without harming transfer.
- Random activity gating trades retention for transfer; averaged over uniform similarity, transfer peaks at gating level 3/4 while retention decreases monotonically.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| High feature similarity and low readout similarity | Vanilla continual learning · Linear teacher-student model | transfer performance | Hurt |
| High feature similarity and low readout similarity | Vanilla continual learning · Linear teacher-student model | retention performance | Hurt |
| Add task-dependent random activity gating | Continual learning · Linear teacher-student model | retention performance | Helped |
| Add task-dependent random activity gating | Continual learning · Linear teacher-student model | transfer performance | Mixed |
| Use moderate random activity gating (ablation) | Continual learning · Uniform task similarity distribution | average transfer and retention performance: α = 3/4 | Helped |
| Use probe-trial-based adaptive activity gating instead of random gating (ablation) | Activity gating · Linear teacher-student model | average transfer performance | Helped |
| Use probe-trial-based adaptive activity gating instead of random gating (ablation) | Activity gating · Permuted MNIST with latent variables | transfer performance | Helped |
| Add task-dependent plasticity gating (ablation) | Continual learning · Over-parameterized linear teacher-student model | transfer and retention performance | No clear effect |
| Add soft-thresholding input activity sparsification (ablation) | Continual learning · Over-parameterized linear teacher-student model | transfer and retention performance: h = √2 erfc⁻¹(α) | No clear effect |
| Regularize weights in the Euclidean metric instead of leaving them unregularized (ablation) | Continual learning · Linear teacher-student model | retention performance | Mixed |
| Use Fisher information metric regularization instead of no weight regularization | Continual learning · Linear teacher-student model | retention performance: Δε_RT = 1 − O(N_s / (N_x (1 − ρ_a²))) | Helped |
| Use Fisher information metric regularization | Continual learning · Linear teacher-student model | transfer performance | No clear effect |
Surprising findings
- Plasticity gating and input soft-thresholding do not affect transfer or retention in the over-parameterized limit.
- Diagonal Fisher approximation is much less robust to task similarity than the exact (layer-wise) Fisher regularization.
Limitations
- Linear projection assumption in the teacher-student model limits direct applicability.
- Diagonal Fisher approximation performs better in the deep network than the teacher-student theory predicts, so theory is not fully matched.
Experimental setup
Datasets: MNIST (permuted, with latent structure)
Benchmarks: permuted MNIST with latent variables linear teacher-student synthetic regression
Metrics: transfer performance (error difference) retention performance (error difference) mean-squared error
Baselines: vanilla model random activity gating Euclidean weight regularization diagonal Fisher approximation
Builds on: elastic weight consolidation activity gating permuted MNIST
Closest papers by meaning
- The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting — An Analytical Model (ICLR 2024 · similarity 0.91)
- Continual Learning with Global Alignment (NeurIPS 2024 · similarity 0.89)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.88)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.88)
- Optimal Task Order for Continual Learning of Multiple Tasks (ICML 2025 · similarity 0.88)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.