Research map / Optimization / Implicit bias and generalization
Deep linear networks for regression are implicitly regularized towards flat minima
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that minimizers of deep linear regression networks have sharpness at least linear in depth, and that gradient flow finds minimizers whose sharpness is within a constant (depending on the data condition number) of this minimum, for both small-scale and residual Gaussian initializations.
Problem
Understanding the sharpness (largest Hessian eigenvalue) of minimizers reached by training deep linear networks on univariate regression, and how it relates to the learning rate threshold for successful training.
Approach
Derives a depth-linear lower bound on the sharpness of any minimizer via a variational characterization of sharpness, and proves that gradient flow from small-scale or residual Gaussian initialization converges to a minimizer whose sharpness is at most a constant times this lower bound, with the constant depending only on the data condition number (not width or depth).
Key results
- Any minimizer of the empirical risk has sharpness lower bounded by approximately 2||w*||^(2-2/L) L a, growing linearly with depth L.
- For small-scale initialization, gradient flow's minimizer satisfies 1 <= S(W_SI)/S_min <= 4 Lambda/lambda (so S <= 4 S_min for white data).
- For residual initialization with Gaussian weights, gradient flow converges globally under a loss-at-initialization condition, and the sharpness ratio satisfies S(W_RI)/S_min <= C Lambda/lambda with C depending only on s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train from small-scale initialization using gradient flow | Deep linear network · Univariate regression with full-rank data covariance | sharpness ratio S(W^SI)/S_min: 1 \leqslant \frac{S(\cW^{\textnormal{SI}})}{S_{\min}} \leqslant 4 \frac{\Lambda}{\lambda} | Helped |
| Train from residual initialization using gradient flow | Deep linear residual network · Univariate regression with full-rank data covariance | sharpness ratio S(W^RI)/S_min: 1 \leqslant \frac{S(\cW^{\textnormal{RI}})}{S_{\min}} \leqslant C \frac{\Lambda}{\lambda} | Helped |
| Increase network depth | Deep linear network minimizers · Empirical-risk minimization | minimum sharpness: S_{\min} \sim 2 \|w^\star\|_2^{2} L a | Hurt |
Surprising findings
- Sharpness at the minimizer found by gradient flow is not arbitrarily small and empirically the critical learning rate for training failure matches 2/S_min rather than depending on initialization scale.
Limitations
- Analysis restricted to deep linear networks and univariate regression, with gradient flow (vanishing learning rate) rather than finite learning rate.
- Small-scale initialization results require the initialization scale to be decreased as depth grows (epsilon = O(1/L^2)), and the residual result requires a bound on initial loss and large-enough depth and width constants.
- Overdetermined setting assumed (full-rank covariance) for the sharpness ratio bounds.
Experimental setup
Datasets: synthetic regression data
Metrics: sharpness (largest Hessian eigenvalue) training risk
Builds on: edge of stability Polyak-Lojasiewicz condition deep linear network balancedness (Du et al. 2018)
Closest papers by meaning
- Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization (ICLR 2026 · similarity 0.91)
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks (ICLR 2019 · similarity 0.90)
- Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks (ICML 2018 · similarity 0.90)
- Implicit Sparse Regularization: The Impact of Depth and Early Stopping (NeurIPS 2021 · similarity 0.90)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.90)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.