Research map / Optimization / Implicit bias and generalization
Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Generalization properties of second-order (Gauss-Newton / natural gradient) optimizers in realistic deep networks are unclear, because prior analyses use simplified models or approximations (Kronecker, damping, constant NTK) that confound the effect of the second-order update.
Approach
Exploits the reversibility of volume-preserving MLP coupling layers to derive a closed-form layer-wise right inverse of the network Jacobian (shown to equal the Moore-Penrose pseudoinverse), giving exact GN updates at roughly SGD-level cost, and proves that any right inverse yields the same loss dynamics to first order in the learning rate.
Key results
- In the full-batch setting (1024 images), GN converges much faster than Adam and SGD on MNIST and CIFAR-10.
- In the mini-batch setting (batch 1024, full datasets), GN saturates early in training and loss stalls, with Adam reaching lower training and test loss; SGD eventually overtakes GN on CIFAR-10.
- NTK and CKA similarity to initialization barely change under GN and SGD, while Adam changes representations, indicating GN operates in the lazy regime.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use exact Gauss-Newton instead of Adam or SGD | RevMLP · MNIST and CIFAR-10, full-batch | training loss progress | Helped |
| Use exact Gauss-Newton instead of Adam | RevMLP · MNIST and CIFAR-10, mini-batch | training and test loss | Hurt |
| Use exact Gauss-Newton instead of SGD | RevMLP · CIFAR-10, mini-batch, longer training | training loss | Hurt |
| Use exact Gauss-Newton instead of SGD or Adam | RevMLP · CIFAR-10, mini-batch | immediate loss decrease on the current mini-batch | Helped |
| Reduce the Gauss-Newton learning rate (ablation) | Exact Gauss-Newton RevMLP · CIFAR-10, mini-batch | training and test loss | No clear effect |
| Add weight decay (ablation) | Exact Gauss-Newton RevMLP · CIFAR-10, mini-batch | performance | No clear effect |
| Use damping instead of truncation for pseudoinverse regularization (ablation) | Exact Gauss-Newton RevMLP · CIFAR-10, mini-batch | training and test performance | Helped |
| Add noise to the matrix before computing its pseudoinverse (ablation) | Exact Gauss-Newton RevMLP · CIFAR-10, mini-batch | training and test performance | Hurt |
| Remove the inverted bottleneck (ablation) | Exact Gauss-Newton RevMLP · CIFAR-10, full-batch and mini-batch | training loss trend | No clear effect |
| Increase the initial weight variance (ablation) | Exact Gauss-Newton RevMLP · CIFAR-10 | training and test loss | Helped |
| Continue training with Gauss-Newton from an Adam-trained initialization (ablation) | RevMLP · CIFAR-10 | training loss | Hurt |
| Continue training with Adam from a Gauss-Newton-trained initialization (ablation) | RevMLP · CIFAR-10 | training loss | Helped |
Surprising findings
- Exact Gauss-Newton overfits each mini-batch strongly yet fails to improve training loss over the long run; GN even increases training loss after about 150 epochs on CIFAR-10.
- Standard regularization (smaller learning rates, weight decay, pseudoinverse damping or noise) does not fix GN's poor generalization.
Limitations
- Finite-learning-rate behavior of the proposed generalized inverse may differ from the Moore-Penrose pseudoinverse update, which is not least-squares equivalent.
- The reversible architecture does not easily accommodate damping or truncation, so the effect of regularized GN at scale is hard to study.
- Experiments are limited to MNIST, CIFAR-10 and small UCI regression tasks with MLP-based reversible models.
Experimental setup
Datasets: MNIST CIFAR-10 UCI Wine UCI Superconductivity
Benchmarks: MNIST CIFAR-10 UCI Wine UCI Superconductivity
Metrics: training loss test loss test accuracy NTK similarity and rate of change CKA similarity percentage change in mini-batch loss
Baselines: Adam SGD
Builds on: Gram-Gauss-Newton natural gradient descent Target Propagation RevNet
Closest papers by meaning
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.90)
- Rethinking Gauss-Newton for learning over-parameterized models (NeurIPS 2023 · similarity 0.90)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.88)
- Theoretical Characterisation of the Gauss Newton Conditioning in Neural Networks (NeurIPS 2024 · similarity 0.88)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.88)
- Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.