Research map / Continual learning / Online and replay-based
A Theory of Initialisation's Impact on Specialisation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically and empirically shows that weight imbalance and high readout-weight entropy at initialisation favour specialised neuron solutions, and uses this to reconcile monotonic vs non-monotonic (Maslow's hammer) forgetting and EWC behaviour in continual learning.
Problem
Prior work attributes non-monotonic catastrophic forgetting vs task similarity to neuron specialisation, but this assumes specialisation occurs; the paper asks what determines whether specialised (localised) representations emerge in neural networks.
Approach
Initialisation, not the activation function, chiefly determines specialisation: weight imbalance between first and last layers (via deep linear network dynamics) and high readout-weight entropy (polar-coordinate readout initialisation) favour specialised solutions, which in turn controls the shape of the forgetting-vs-similarity curve and EWC effectiveness.
Key results
- In the deep linear network, specialisation emerges when the faster pathway has larger weight imbalance than the slower one (phase diagram with clear transition).
- In beta-VAE on 3DShapes, smaller Xavier gain (0.3) yields higher DCI disentanglement scores than gain 2 at similar reconstruction loss.
- In teacher-student continual learning, non-specialised (symmetric or large asymmetric second-head) initialisations give monotonic forgetting vs task similarity, while specialised ones give the non-monotonic Maslow's hammer profile.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the fast pathway's weight imbalance relative to the slow pathway | Deep linear network · Synthetic regression | specialisation | Helped |
| Use smaller Xavier gain, with gain g in the first encoder block and 1/g in the readout | β-VAE · 3DShapes | DCI scores | Helped |
| Use smaller Xavier gain, with gain g in the first encoder block and 1/g in the readout | β-VAE · 3DShapes | reconstruction loss | No clear effect |
| Avoid specialisation in the student | Two-task teacher-student continual learning · Synthetic tasks across task similarities | forgetting profile | Mixed |
| Use large asymmetric initialisation for the second-task readout weights | Two-task teacher-student continual learning · Synthetic tasks across task similarities | forgetting profile | Mixed |
| Add specialisation to the first-task solution | EWC · Two-task teacher-student continual learning | generalisation error on both tasks | Helped |
| Remove specialisation from the first-task solution | EWC · Two-task teacher-student continual learning | generalisation error on both tasks | Hurt |
| Increase asymmetry in the readout weights | Teacher-student network · Continual learning | specialisation | Helped |
Surprising findings
- ReLU students can specialise while sigmoidal (erf-type) students do not always specialise, contradicting the prior view that activation function determines specialisation.
- EWC is largely ineffective when the student does not specialise, since it cannot distinguish redundant nodes and either under- or over-regularises.
Limitations
- Analysis relies on simplified Gaussian data and teacher-student setups, far from real-world data.
- Disentanglement experiments are limited (one dataset, 3DShapes, 4 seeds, 20 epochs) and only preliminary validation.
- Deep linear analysis assumes alignment of pathway weights to dataset singular vectors (silent alignment) and mutually diagonalisable correlation matrices.
Experimental setup
Datasets: 3DShapes MNIST synthetic Gaussian teacher-student data
Benchmarks: 3DShapes disentanglement (DCI) teacher-student generalisation error MNIST continual learning
Metrics: DCI disentanglement score DCI completeness and informativeness generalisation error forgetting specialisation entropy
Baselines: elastic weight consolidation standard Xavier initialisation (gain 1)
Builds on: Maslow's hammer continual learning analysis deep linear network dynamics silent alignment effect elastic weight consolidation beta-VAE
Closest papers by meaning
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.87)
- Disentangling and mitigating the impact of task similarity for continual learning (NeurIPS 2024 · similarity 0.86)
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.86)
- Flexible task abstractions emerge in linear networks with fast and bounded units (NeurIPS 2024 · similarity 0.85)
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.85)
- How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.