Research map / Probabilistic and scientific modeling / Dynamical systems
Critical Learning Periods Emerge Even in Deep Linear Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that critical learning periods arise in deep linear networks, analytically tractable models, with effects driven by depth and data structure through multi-pathway competition and matrix-completion transfer.
Problem
Why critical learning periods emerge in deep networks and whether they depend on architecture or optimization details, or on more fundamental factors.
Approach
Uses deep linear networks, stripped of nonlinearities and biology, as a minimal analytically tractable model: a two-pathway model with gating deficits and a deep matrix-factorization model for matrix completion with pre-training, deriving ODEs for singular value dynamics that match SGD.
Key results
- Early gating deficit to one pathway leads the other pathway to learn all features, with the effect growing with depth (depths 2 to 13 shown); late deficits have negligible effect.
- In matrix completion with depth 3, pre-training on a higher-rank task (rank 10) before a rank 5 task impairs final generalization in low-sample regimes, while low-to-high rank transfer shows no such effect.
- Analytical ODEs for singular value dynamics match SGD simulations before, during and after deficits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply a temporary gating deficit early to one pathway | Multi-pathway deep linear network · Feature learning | pathway-specific feature learning | Helped |
| Apply a temporary gating deficit late rather than early | Multi-pathway deep linear network · Feature learning | pathway-specific feature learning | No clear effect |
| Increase network depth under an early pathway deficit (ablation) | Multi-pathway deep linear network · Feature learning | effect of temporary deficit | Hurt |
| Apply a middle-training deficit to one pathway | Multi-pathway deep linear network · Feature learning | feature learning | Mixed |
| Lesion a feature in the otherwise normal pathway during deprivation (ablation) | Multi-pathway deep linear network · Feature learning | learning of the lesioned feature by the deprived pathway | Helped |
| Pre-train on a higher-rank task before a lower-rank final task | Deep linear matrix completion · Final-task generalization | reconstruction error | Hurt |
| Pre-train on a lower-rank task before a higher-rank final task (ablation) | Deep linear matrix completion · Final-task generalization | reconstruction error | No clear effect |
| Pre-train on a rank 2 or rank 5 task before a rank 5 final task (ablation) | Deep linear matrix completion · Final-task generalization | reconstruction error | No clear effect |
| Increase pre-training duration on a different task (ablation) | Deep linear matrix completion · Final-task generalization | reconstruction error | Hurt |
| Increase the depth of the matrix-factorization network (ablation) | Deep linear matrix completion · Final-task generalization after pre-training | reconstruction error | Hurt |
| Use depth 1 instead of a deeper matrix-factorization parameterization (ablation) | Deep linear matrix completion · Final-task generalization | sensitivity to initial task | No clear effect |
| Increase the number of observed entries in the matrix-completion tasks (ablation) | Deep linear matrix completion · Final-task generalization | sensitivity to initial task | Helped |
Surprising findings
- Critical periods appear in deep linear networks, which have no nonlinearities or biochemical plasticity changes.
- Deficits affecting learning in matrix completion vanish when the number of observations is large (N=3000 or 4000).
Limitations
- Deep linear networks lack nonlinearities, so results may not transfer directly to nonlinear DNNs (only partially checked with Tanh and ReLU in the multi-pathway setting).
- Analysis relies on balanced/small initialization and distinct nonzero singular values for the exact ODEs.
Experimental setup
Datasets: synthetic hierarchical task (whitened inputs, 8-dim one-hot inputs, 15-dim outputs) synthetic 100x100 low-rank matrices with 1750-5000 observed entries
Metrics: singular value trajectories average squared reconstruction error per entry agreement between ODE and SGD dynamics
Builds on: deep linear networks (Saxe et al.) implicit bias of gradient descent in deep matrix factorization multi-pathway deep linear networks
Closest papers by meaning
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.85)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.85)
- A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language (ICLR 2025 · similarity 0.84)
- Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks (NeurIPS 2025 · similarity 0.83)
- Flexible task abstractions emerge in linear networks with fast and bounded units (NeurIPS 2024 · similarity 0.83)
- Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.