Research map / Probabilistic and scientific modeling / Dynamical systems
Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that layerwise linear models (diagonal and deep linear networks) should be solved first, using a dynamical feedback principle to explain emergence, neural collapse, lazy/rich regimes, and grokking.
Problem
Understanding dynamical phenomena in deep neural networks (emergence, neural collapse, lazy/rich regimes, grokking) is hard because non-linear networks are difficult to analyze; the paper asks whether solvable layerwise linear models can capture the core dynamics.
Approach
Proposes the dynamical feedback principle: in layerwise models, the magnitude of one layer scales the gradient of the others, producing amplifying (rich-get-richer) dynamics and sigmoidal, stage-like learning that unify several DNN phenomena, including ones with non-linear activations.
Key results
- Reducing the weight-to-target ratio (e.g., small initialization, target upscaling, output rescaling) removes the grokking generalization delay in a 4-layer tanh MLP trained on 1000 MNIST samples.
- The toy 2-layer model yields sigmoidal (delayed) saturation of a_i b_i versus immediate exponential saturation in the linear model, explaining stage-like learning.
- Layer imbalance (lambda-balanced condition) controls transition between rich and lazy dynamics in linear networks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use upstream initialization to emphasize learning in earlier layers | convolutional neural networks · feature learning and interpretability | feature learning | Helped |
| use upstream initialization | transformer · modular arithmetic task | generalization delay | Helped |
| reduce the weight-to-target ratio | neural networks in grokking settings · grokking | generalization delay | Helped |
Surprising findings
- Grokking may be explained as a special case of a poorly initialized lazy regime rather than an artifact of algorithmic datasets.
Limitations
- Layerwise linear models lack non-linear activations, so some phenomena may require non-linearity beyond their explanatory reach.
- Many results rely on small/balanced initialization or whitened inputs and gradient flow assumptions.
- Analyses are mostly intuitive derivations and rely on prior work for rigor.
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: training and test accuracy (grokking delay) NTK change
Baselines: linear model (lazy dynamics) NTK initialization
Builds on: Saxe et al. deep linear networks dynamics Kunin et al. layer imbalance Nam et al. emergence from dynamics Mixon et al. unconstrained feature model
Closest papers by meaning
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.86)
- Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking (NeurIPS 2025 · similarity 0.84)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.84)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.83)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.83)
- Grokking as a First Order Phase Transition in Two Layer Networks (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.