Research map / Architectures and efficiency / RNNs and normalization
Theoretical Characterisation of the Gauss Newton Conditioning in Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives tight upper bounds on the condition number of the Gauss-Newton matrix for deep linear networks, residual linear networks, and one-hidden-layer (Leaky)ReLU networks, validated empirically on MNIST and CIFAR-10.
Problem
Understanding how architectural components (depth, width, skip connections, normalization, data covariance) shape the conditioning of the Gauss-Newton matrix, which governs optimization difficulty and preconditions adaptive methods, has been largely unexplored theoretically.
Approach
Expresses the GN matrix of deep linear networks as a Kronecker-structured sum over layers, then bounds its extreme non-zero eigenvalues with Weyl's inequalities to get a convex-combination bound on the condition number; for residual networks the skip connection shifts singular values by beta, which provably lowers the bound.
Key results
- For deep linear networks at fixed width, the GN condition number grows quadratically with depth, and scaling width proportionally with depth keeps it controlled.
- The derived convex-combination upper bound is tight and tracks the GN condition number at initialization and throughout training (3-layer linear net, condition number ranging about 6 to 12).
- Skip connections (residual linear networks) provably reduce the condition-number bound, and batch normalization empirically improves GN conditioning on non-whitened CIFAR-10 data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Whiten the input data | Neural networks · Whitened versus unwhitened data | condition number | Helped |
| Increase depth while holding hidden width constant | Linear networks · Whitened MNIST | condition number | Hurt |
| Increase hidden-layer width | Linear networks · Linear-network conditioning experiments | condition number | Helped |
| Scale hidden width proportionally with depth | Linear networks · Linear-network conditioning experiments | condition number | Helped |
| Add skip connections | Linear networks · Residual-network conditioning experiments | condition number | Helped |
| Increase the skip-connection weight β | Residual linear networks · Residual-network conditioning experiments | condition number | Helped |
| Use a lower Leaky ReLU negative slope | One-hidden-layer networks · Leaky ReLU versus linear activation | condition number | Helped |
| Use ReLU activation instead of a linear activation | One-hidden-layer networks · One-hidden-layer activation comparison | condition number | Helped |
| Add batch normalization | One-hidden-layer linear network · Downsampled CIFAR-10 (d=64, n=1000) | condition number | Helped |
| Increase the number of filters | Two-layer linear CNN · Whitened MNIST subsample (n=1000) | condition number | Hurt |
| Increase kernel size | Two-layer linear CNN · Whitened MNIST subsample (n=1000) | condition number | Helped |
| Prune more weights at initialization | ResNet20 · CIFAR-10 subset (n=1000) | condition number | Hurt |
Surprising findings
- Naively taking the maximum over per-layer terms yields a vacuous bound; the convex-combination weighting is essential because of a self-balancing effect across layers.
- For one-hidden-layer Leaky ReLU networks the condition number increases with width, contrary to the linear case.
Limitations
- Bounds are agnostic to training dynamics and cannot distinguish random initializations from trained solutions.
- Non-linear analysis is restricted to two-layer (one-hidden-layer) networks; normalization and transformers are not analyzed analytically.
- Analysis focuses on the Gauss-Newton term G_O; the functional Hessian term is only treated in an appendix for one-hidden-layer linear networks.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: condition number of Gauss-Newton matrix training loss
Builds on: Gauss-Newton decomposition deep linear network analysis (Saxe et al.) Weyl's inequality K-FAC
Closest papers by meaning
- Rethinking Gauss-Newton for learning over-parameterized models (NeurIPS 2023 · similarity 0.90)
- Towards a Mathematical Understanding of the Difficulty in Learning With Feedforward Neural Networks (CVPR 2018 · similarity 0.88)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.88)
- SGD vs GD: Rank Deficiency in Linear Networks (NeurIPS 2024 · similarity 0.85)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.85)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.