Research map / Learning theory and algorithms / Matrix and tensor methods
Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a self-consistent matrix ODE for shallow linear networks under gradient flow that interpolates between lazy and balanced dynamics via a mixed regime, and uses it to give a phase diagram for MSE low-rank recovery.
Problem
Understanding how training dynamics of linear networks transition between the lazy (NTK) regime and the active/balanced regime as a function of width and initialization variance, and what lies between them.
Approach
Shows that for i.i.d. Gaussian initialization of width w, the learned matrix obeys a self-consistent dynamics with preconditioners sqrt(AA^T + sigma^4 w^2 I) and sqrt(A^TA + sigma^4 w^2 I); singular values below sigma^2 w evolve lazily and those above evolve in balanced fashion, giving a mixed regime with low-rank bias and guaranteed convergence.
Key results
- Self-consistent dynamics closely match gradient descent test error and singular value trajectories in the active regime (gamma_sigma2=-1.85, gamma_w=2.25, rank 5).
- Theory gives a phase diagram: lazy regime when gamma_sigma2+gamma_w>1, active/mixed regime when <1, with recovery of A* in the active regime.
- Proves Theorem bounding ||C_1 - hat C_1||_op by min{O(sigma^2 w), O(sqrt(d/w)||C_1||_op)} for general convex costs under gradient flow with high probability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mixed/active training instead of pure lazy training | Shallow linear network · Low-rank matrix recovery | test error | Helped |
| Use mixed dynamics rather than balanced dynamics from random initialization | Shallow linear network · Low-rank matrix recovery | recovery success | Helped |
| Use mixed dynamics rather than lazy dynamics | Shallow linear network · Low-rank matrix recovery | low-rank recovery | Helped |
| Use self-consistent mixed dynamics instead of gradient descent | Shallow linear network · Active-regime low-rank matrix recovery | train and test error | No clear effect |
| Increase the distance into the active regime | Shallow linear network · Low-rank matrix recovery | iterations until minimal test error | Hurt |
Surprising findings
- Lazy regime for finite variance only appears for gamma_w>2 in the sparse case, suggesting it is rarely observed in practice.
- The optimal test error occurs in a short window where large singular values have converged but the rest have not, i.e. early stopping is beneficial.
Limitations
- Theory relies on overparametrization (w >> d) and a sigma^4 w^2 small condition; the underparametrized regime is only explored empirically.
- Approximation error does not vanish; results are approximate and high-probability rather than exact.
- The phase diagram theorem applies only to gamma_w>1 and the MSE task with low-rank target.
Experimental setup
Datasets: synthetic low-rank matrix with Gaussian noise
Benchmarks: synthetic low-rank matrix recovery
Metrics: test error (normalized Frobenius) stable rank iterations to minimal test error
Baselines: lazy dynamics balanced dynamics gradient descent
Builds on: balanced dynamics neural tangent kernel Marchenko-Pastur law
Closest papers by meaning
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.89)
- The Dynamics of Learning: A Random Matrix Approach (ICML 2018 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.87)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.87)
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.