Research map / Optimization / Implicit bias and generalization
Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a dynamical mean field theory for gradient descent in randomly initialized deep linear networks trained on random data, capturing width, depth, data quantity, SGD, residual scaling, and power-law dynamics.
Problem
Characterizing training dynamics from random initialization of deep networks as a function of width, depth, dataset size, parameterization (NTK vs muP), and learning-rate transfer, which existing theories do not capture jointly.
Approach
A dynamical mean field theory specialized to deep linear networks averages over both random initial weights and random training data (full-batch or online SGD) in a joint proportional limit (P, D, N large), yielding a closed set of correlation/response equations whose size does not grow with D, N, P; extended to residual networks with branch scaling 1/sqrt(depth) for an infinite-depth limit.
Key results
- Theory shows NTK-parameterized networks shift optimal learning rate with width while muP networks keep it approximately constant, matching simulations.
- For residual networks with branch scaling beta0/sqrt(L), the dynamics converge to a well-defined infinite-depth limit, and learning rate transfer across depth is preserved.
- For power-law structured data, the theory reproduces accelerated loss scaling t^{-2beta/(1+beta)} for hard tasks (beta<1) in deep linear networks, versus t^{-beta} for easy tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase network width under μP/MFP parameterization | Deep linear networks · Random isotropic data, full-batch GD | training convergence rate | Helped |
| Increase network width under NTK parameterization | Deep linear networks · Random isotropic data, full-batch GD | training convergence rate | Hurt |
| Use μP instead of NTK parameterization | Deep linear networks · Random isotropic data, varying width | optimal learning-rate transfer | Helped |
| Increase width under NTK parameterization | Deep linear networks · Random isotropic data, varying width | optimal learning-rate transfer | Hurt |
| Increase dataset size when training is repeated for many iterations | Deep linear networks · Random isotropic data, full-batch GD | test loss | Mixed |
| Increase online SGD batch size | Deep linear networks · Random isotropic data, online SGD | training convergence rate | Helped |
| Increase network width | Deep linear networks · Random isotropic data, online SGD | test loss | Helped |
| Scale the residual branch by 1/√L | Residual networks · Varying network depth | optimal learning-rate transfer | Helped |
| Keep the residual branch scale constant as depth increases | Residual networks · Varying network depth | optimal learning-rate transfer | Hurt |
| Increase richness parameter γ₀ | Deep residual networks · Random isotropic data, large depth | training convergence rate | Helped |
| Increase residual branch multiplier β₀ | Deep residual networks · Random isotropic data, large depth | initial kernel | Helped |
| Use feature learning instead of lazy learning | Deep linear networks · Power-law data, hard tasks (β<1) | test-loss convergence exponent | Helped |
Surprising findings
- Wider NTK-parameterized networks can train more slowly, and at fixed learning rate finite-width effects slow the dynamics.
Limitations
- Limited to deep linear networks, so nonlinear function approximation is not captured.
- Residual DMFT cost grows with depth (~L^2 response functions) and training steps T, making deep networks expensive.
- Power-law SGD results are approximate for small width and batch size and assume gradient flow, ignoring discrete-time edge-of-stability effects.
Experimental setup
Datasets: random Gaussian isotropic data power-law covariance synthetic data
Metrics: train loss test loss
Baselines: NTK parameterization direct simulation of finite-width linear networks
Builds on: dynamical mean field theory muP / maximal update parameterization neural tangent kernel depth-muP
Closest papers by meaning
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.91)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.90)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.90)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.90)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.90)
- Generalization of Scaled Deep ResNets in the Mean-Field Regime (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.