Research map / Learning theory and algorithms / Matrix and tensor methods
Practical Gauss-Newton Optimisation for Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KFRA instead of tuned ADAM | Deep autoencoder training · CURVES, FACES, and MNIST | training error | Helped |
| Use second-order optimisation instead of ADAM | Deep autoencoder training · FACES | optimisation stability | Helped |
| Use KFRA instead of KFAC | Deep autoencoder training · CURVES, FACES, and MNIST | optimisation performance | Helped |
| Use second-order optimisation instead of ADAM | MNIST even/odd mixture classifier · MNIST even/odd mixture | final objective value | Hurt |
| Use second-order optimisation instead of ADAM | MNIST even/odd mixture classifier · MNIST even/odd mixture | wall-clock time | Mixed |
| Use KFLR instead of KFAC | MNIST even/odd mixture classifier · MNIST even/odd mixture | optimisation performance | No clear effect |
| Add moving-average momentum to curvature matrices (ablation) | KFRA and KFAC · CURVES, FACES, and MNIST | training error | No clear effect |
| Add momentum to parameter updates (ablation) | Second-order methods · Training experiments | optimisation stability | Hurt |
| Use default rather than tuned damping settings | Second-order methods · CURVES, FACES, and MNIST | optimisation performance | Helped |
Closest papers by meaning
- Towards a Mathematical Understanding of the Difficulty in Learning With Feedforward Neural Networks (CVPR 2018 · similarity 0.90)
- Theoretical Characterisation of the Gauss Newton Conditioning in Neural Networks (NeurIPS 2024 · similarity 0.88)
- Practical Quasi-Newton Methods for Training Deep Neural Networks (NeurIPS 2020 · similarity 0.88)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.87)
- A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks (NeurIPS 2024 · similarity 0.86)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.