Research map / Optimization / Optimizers
A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LNGD, a layer-wise natural gradient optimizer that approximates each layer's Fisher block as a Kronecker product of a dense factor and a diagonal factor (sampled per layer without full backprop) plus an adaptive layer-wise learning rate, evaluated on CIFAR-10, ImageNet and WMT transformer training.
Problem
Natural gradient methods are effective for training deep networks but their computational and storage cost of inverting Fisher information matrices is too high for practical large-scale use.
Approach
Approximates each layer's Fisher block as a Kronecker product of a dense factor and a diagonal factor, computing the factors via layer-wise sampling from Gaussian-assumed layer predictive distributions so no full backpropagation is needed for curvature, with trace-preserving scaling and an adaptive per-layer learning rate derived from a quadratic model.
Key results
- On CIFAR-10 with ResNet-18, LNGD reaches 91% top-1 accuracy in 36 epochs and 189.69 s, versus 45 epochs and 241.86 s for KFAC.
- On ImageNet with ResNet-50, LNGD reaches 75.9% top-1 in 36 epochs and 6.46 h, versus 39 epochs and 8.02 h for KFAC.
- On WMT Transformer-Big, LNGD reaches 32% BLEU in 19K steps and 2.85 h, versus 22K steps and 3.75 h for KFAC and 25K steps and 3.39 h for ADAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LNGD instead of KFAC | ResNet-18 · CIFAR-10 | total time to 91% test accuracy: 241.86s -> 189.69s | Helped |
| Use LNGD instead of KFAC | ResNet-50 · ImageNet-1K | total time to 75.9% test accuracy: 8.02h -> 6.46h | Helped |
| Use LNGD instead of KFAC | Transformer-Big · WMT English-German | steps to BLEU 32%: 22K -> 19K | Helped |
| Use LNGD instead of KFAC | Transformer-Big · WMT English-German | best test BLEU: 33.45% -> 33.55% | Helped |
| Use LNGD instead of SGD | Transformer-Big · WMT English-German | best test BLEU: 31.87% -> 33.55% | Helped |
| Add sampling approximation of the Fisher information matrix (ablation) | LNGD on ResNet-50 · ImageNet-1K | total time to 75.9% test accuracy: 7.43h -> 6.46h | Helped |
| Add sampling approximation of the Fisher information matrix (ablation) | LNGD on ResNet-50 · ImageNet-1K | best test accuracy: 76.50% -> 76.73% | Helped |
| Add adaptive layer-wise learning rate (ablation) | LNGD on ResNet-50 · ImageNet-1K | total time to 75.9% test accuracy: 7.06h -> 6.46h | Helped |
| Add adaptive layer-wise learning rate (ablation) | LNGD on ResNet-50 · ImageNet-1K | best test accuracy: 76.57% -> 76.73% | Helped |
Surprising findings
- The variant without the sampling approximation (LNGD-lr, exact Fisher per epoch) converged in fewer epochs but took about 15% more time than LNGD.
Limitations
- Convergence theory is established only for a two-layer ReLU network with fixed second-layer weights and an NGD-style update, not for the full LNGD algorithm on deep networks.
- Relies on a Gaussian distribution assumption for layer activations, supported only by empirical histograms on ResNet-18.
- Hyperparameters are tuned by grid/random search and results averaged over 5 runs; no separate code or data release is stated.
Experimental setup
Datasets: CIFAR-10 ImageNet-1K WMT English-German
Benchmarks: CIFAR-10 ImageNet-1K WMT English-German
Metrics: top-1 test accuracy training loss epochs to 91% accuracy wall-clock time BLEU
Baselines: SGD ADAM KFAC EKFAC TKFAC NG+
Builds on: KFAC TKFAC EKFAC natural gradient descent
Closest papers by meaning
- Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks (NeurIPS 2020 · similarity 0.90)
- Fast Convergence of Natural Gradient Descent for Over-Parameterized Neural Networks (NeurIPS 2019 · similarity 0.89)
- An Improved Empirical Fisher Approximation for Natural Gradient Descent (NeurIPS 2024 · similarity 0.89)
- Practical Quasi-Newton Methods for Training Deep Neural Networks (NeurIPS 2020 · similarity 0.88)
- Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC (ICML 2024 · similarity 0.86)
- Near-optimal Sketchy Natural Gradients for Physics-Informed Neural Networks (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.