Research map / Optimization / Optimizers
Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| track diagonal second moments in the Kronecker-factored eigenbasis (EKFAC) instead of KFAC's Kronecker-product eigenvalues | KFAC · MNIST deep auto-encoder | training loss per epoch | Helped |
| running-average scaling (EKFAC-ra) instead of KFAC | KFAC · MNIST deep auto-encoder | training loss vs wall-clock time | Helped |
| EKFAC instead of KFAC | KFAC · MNIST deep auto-encoder | validation loss | No clear effect |
| amortize eigendecomposition with EKFAC instead of amortized KFAC (ablation) | KFAC · MNIST deep auto-encoder | training loss per epoch | Helped |
| recompute inverse/eigenbasis every update vs amortized (ablation) | KFAC · MNIST deep auto-encoder | training loss per epoch | Helped |
| EKFAC (amortized KFE) instead of KFAC | KFAC · VGG11 CIFAR-10 | training loss per epoch | Helped |
| EKFAC instead of KFAC with batch size 200 | KFAC · VGG11 CIFAR-10 | wall-clock time | Mixed |
| EKFAC-ra instead of KFAC and SGD momentum with BN | KFAC · ResNet34 CIFAR-10 | training loss per epoch and compute time | Helped |
| EKFAC instead of KFAC | KFAC · CIFAR-10 networks | validation accuracy | No clear effect |
Closest papers by meaning
- A Trace-restricted Kronecker-Factored Approximation to Natural Gradient (AAAI 2021 · similarity 0.89)
- A Kronecker-factored approximate Fisher matrix for convolution layers (ICML 2016 · similarity 0.89)
- Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.87)
- Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC (ICML 2024 · similarity 0.87)
- Provable Acceleration of Nesterov's Accelerated Gradient for Asymmetric Matrix Factorization and Linear Neural Networks (NeurIPS 2024 · similarity 0.87)
- An Improved Empirical Fisher Approximation for Natural Gradient Descent (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.