Research map / Architectures and efficiency / RNNs and normalization
From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives exact gradient-flow solutions for two-layer deep linear networks under lambda-balanced initializations, interpolating between rich and lazy learning regimes and illustrating applications to continual, reversal, transfer learning and fine-tuning.
Problem
Understanding how initialization (absolute and relative weight scale between layers) controls whether deep networks learn in a rich (feature-learning) or lazy (kernel) regime, which lacks clear exact theoretical predictions for wide networks.
Approach
Relaxes the zero-balanced initialization assumption of prior exact solutions to lambda-balanced initializations (W2^T W2 - W1 W1^T = lambda I), yielding closed-form Riccati-based solutions for the dynamics of QQ^T, representational similarity, and finite-width NTK across the full rich-to-lazy spectrum, including unequal input/output dimensions.
Key results
- Derives closed-form solution (Theorem 1) for QQ^T dynamics under lambda-balanced initialization, valid for unequal input/output dimensions without invertibility assumptions on B.
- Shows singular value transition function converges to sigmoidal (lambda->0) and exponential (lambda->±inf) forms, interpolating rich to lazy regimes.
- Finds lambda=0 networks are most robust to parameter noise, and large positive lambda yields structured lazy representations that aid transfer learning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Set relative scale λ to 0 instead of large |λ| | Two-layer linear network with task-aligned initialization · Learning dynamics | learning-curve regime | Helped |
| Increase the magnitude of relative scale |λ| | Two-layer linear network · Learning dynamics | learning-curve regime | Mixed |
| Use λ = 0 instead of large positive or negative λ | Two-layer linear network with task-agnostic initialization · Hierarchical semantic learning | task-specific representation | Helped |
| Use large positive λ | Two-layer linear network · Hierarchical semantic learning | input representation task specificity | Hurt |
| Use large negative λ | Two-layer linear network · Hierarchical semantic learning | input representation task specificity | Hurt |
| Set λ to 0 rather than a nonzero value | Two-layer linear network with equal input-output dimensions · Parameter-noise robustness | expected loss under parameter noise | Helped |
| Use λ approaching positive infinity | Funnel network · NTK dynamics | NTK movement from initialization | Hurt |
| Use λ approaching negative infinity | Inverted-funnel network · NTK dynamics | NTK movement from initialization | Hurt |
| Use λ approaching zero | Funnel, square, and inverted-funnel networks · Learning dynamics | rich-regime behavior | Helped |
| Use nonzero λ instead of zero-balanced initialization | Two-layer linear network · Reversal learning | reversal-learning success | Helped |
| Train on subsequent tasks | Two-layer linear network with any λ · Continual learning | retention of previously acquired knowledge | Hurt |
| Use λ = 0 rather than a lazy input representation | Two-layer linear network pretrained on the hierarchical semantic task · Transfer learning to a new feature | hierarchical generalization | Helped |
Surprising findings
- Reversal learning in zero-balanced deep linear networks can be catastrophically slowed near a saddle point, while non-zero lambda avoids it.
Limitations
- Restricted to whitened inputs and two-layer deep linear networks; nonlinear extension not established.
- Assumes lambda-balanced initialization, which holds only approximately for wide random initializations.
- Fine-tuning and LoRA connections are discussed but not explored empirically.
Experimental setup
Datasets: hierarchical semantic learning task (synthetic)
Metrics: training loss representational similarity matrix NTK kernel distance from initialization expected loss under input and parameter noise
Baselines: zero-balanced (lambda=0) deep linear network solutions LeCun initialization Gaussian large-variance (lazy) initialization
Builds on: Saxe et al. exact solutions for deep linear networks matrix Riccati formulation (Braun et al.) neural tangent kernel Saxe et al. 2019 implicit bias / balancedness representational similarity analysis
Closest papers by meaning
- Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning (NeurIPS 2024 · similarity 0.92)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.90)
- Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes (NeurIPS 2024 · similarity 0.89)
- How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.89)
- The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks (NeurIPS 2026 · similarity 0.89)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.