Research map / Continual learning / Online and replay-based
Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes representation discrepancy, a linear-alignment metric for representation forgetting in continual learning, derives an analytic upper bound showing forgetting saturates, grows with layer depth, and slows with width, and validates on Split-CIFAR100 and ImageNet1K with ResNets.
Problem
Representation forgetting (loss of hidden-layer features for earlier tasks during continual learning) has been measured empirically but lacks theoretical analysis; existing metrics such as linear-probe accuracy drop are hard to analyze mathematically.
Approach
Define representation discrepancy as the minimum, over linear transformations, of the worst-case distance between the k-th layer representation spaces of the model before and after learning additional tasks; bound it analytically using perturbation analysis of sequential weight updates, yielding a two-phase (forgetting then saturation) curve whose asymptote scales with layer representation norm an
Key results
- Representation discrepancy correlates linearly with representation forgetting, with R^2 of 0.88 on Split-CIFAR100 and 0.74 on ImageNet1K.
- Layer-wise forgetting (linear probe drop for task 1) shows a strong linear relationship with the representation norm (R^2 0.84 on ImageNet32), and the representation norm grows linearly with layer index.
- Saturation time Delta t_sat decreases with layer index and increases with channel width on Split-CIFAR100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Measure representation forgetting with representation discrepancy | Continual learning representations · Split-CIFAR100 | Correlation with representation forgetting (R²): R² = 0.88 | Helped |
| Measure representation forgetting with representation discrepancy | Continual learning representations · ImageNet1K | Correlation with representation forgetting (R²): R² = 0.74 | Helped |
| Increase the layer index | ResNet continual learning · Split-CIFAR100 | Representation forgetting | Hurt |
| Increase network width | ResNet continual learning · Split-CIFAR100 | Representation-forgetting convergence rate | Helped |
| Increase the number of subsequently learned tasks | Task-1 representations at block 9 · Split-CIFAR100 | Linear-probing representation forgetting: Δt=10 | Mixed |
| Increase the number of subsequently learned tasks | Task-1 representations at block 9 · ImageNet1K | Linear-probing representation forgetting: Δt=10 | Mixed |
| Use representation-space size as a predictor of representation forgetting | ResNet continual learning · ImageNet1K | Correlation with representation forgetting (R²): R² = 0.84 | Helped |
Surprising findings
- The layer-wise representation norm is strongly linear in layer index, so asymptotic forgetting is predicted to grow linearly with depth.
Limitations
- The key assumption that a linear map aligns weight matrices across tasks is only empirically supported and may not hold for all architectures.
- The linearity of the lower bound on the distance between representation spaces (Delta t dependence) is assumed rather than proven.
- Experiments are limited to small ResNets on downsampled image datasets.
Experimental setup
Datasets: Split-CIFAR100 ImageNet1K (downsampled, ImageNet32)
Benchmarks: Split-CIFAR100 ImageNet1K
Metrics: linear probing accuracy drop representation discrepancy saturation time (Delta t_sat)
Baselines: linear probing accuracy drop (representation forgetting) continual learning error metric (output-space distance)
Builds on: Lee et al. 2024 continual learning error bound (perturbation analysis) linear probing representation forgetting layer cushion activation contraction
Closest papers by meaning
- Understanding the Forgetting of (Replay-based) Continual Learning via Feature Learning: Angle Matters (ICML 2025 · similarity 0.90)
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.90)
- Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks (ICML 2025 · similarity 0.89)
- The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting — An Analytical Model (ICLR 2024 · similarity 0.89)
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.89)
- Do Your Best and Get Enough Rest for Continual Learning (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.