Research map / Optimization / Optimizers
Tractable structured natural-gradient descent using local parameterizations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use structured NGD with Kronecker-structured precision updates instead of Adam | CNN · STL-10 | test accuracy: 69.5% -> 75.8% | Helped |
| Use structured NGD with Kronecker-structured precision updates instead of Adam | CNN · CIFAR-10 | test accuracy: 82.3% -> 85.0% | Helped |
| Use structured NGD with Kronecker-structured precision updates instead of Adam | CNN · CIFAR-100 | test accuracy: 53.3% -> 55.2% | Helped |
| Update both Wishart parameters rather than only B using re-parameterizable gradients (ablation) | Wishart search update · Metric nearness | solution quality | Helped |
| Evaluate gradients at the mean (ablation) | Wishart search update · Metric nearness | convergence | No clear effect |
| Use upper-triangular rather than diagonal precision structure (ablation) | Gaussian mixture approximation · 80-Dim mixture of Student's t distributions | approximation quality | Helped |
| Use upper-triangular rather than lower-triangular precision structure (ablation) | Gaussian mixture approximation · 80-Dim mixture of Student's t distributions | approximation quality | Helped |
| Use upper-triangular precision structure instead of full precision (ablation) | Gaussian mixture approximation · 80-Dim mixture of Student's t distributions | approximation quality | No clear effect |
| Use a lower rather than upper precision structure (ablation) | Structured second-order optimization · Rosenbrock and Dixon-Price, 200-Dim | convergence | Helped |
| Use lower Heisenberg precision structure instead of BFGS or Adam | Structured second-order optimization · Rosenbrock and Dixon-Price, 200-Dim | convergence | Helped |
Closest papers by meaning
- Exact natural gradient in deep linear networks and its application to the nonlinear case (NeurIPS 2018 · similarity 0.88)
- Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC (ICML 2024 · similarity 0.87)
- Improving Computational Complexity in Statistical Models with Local Curvature Information (ICML 2024 · similarity 0.86)
- An Improved Empirical Fisher Approximation for Natural Gradient Descent (NeurIPS 2024 · similarity 0.86)
- A geometric framework for momentum-based optimizers for low-rank training (NeurIPS 2025 · similarity 0.86)
- A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.