Research map / Optimization / Optimizers
A Kronecker-factored approximate Fisher matrix for convolution layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KFC-pre approximate natural-gradient preconditioning instead of SGD with momentum | CIFAR-10 convolutional network · CIFAR-10 | time to 19% test error: 9 minutes -> 3 minutes | Helped |
| Use KFC-pre approximate natural-gradient preconditioning instead of SGD with momentum | CIFAR-10 convolutional network · CIFAR-10 | time to 6% training error: 30 minutes -> 4 minutes | Helped |
| Use KFC-pre approximate natural-gradient preconditioning instead of SGD with momentum | SVHN convolutional network · SVHN | time to 2.78% test error: 120 minutes; SGD did not reach it within 250 minutes | Helped |
| Use KFC-pre instead of SGD with momentum at mini-batch size 4096 | CIFAR-10 convolutional network · CIFAR-10 | updates to 10% training error: 6000 updates -> 300 updates | Helped |
| Replace max-pooling with average pooling (ablation) | CIFAR-10 and SVHN convolutional networks · CIFAR-10 and SVHN | spatial and cross-map derivative correlations | Hurt |
| Add batch normalization to the network (ablation) | CIFAR-10 convolutional network · CIFAR-10 | test error before overfitting | Helped |
| Add batch normalization to the network (ablation) | CIFAR-10 convolutional network · CIFAR-10 | training optimization speed | Hurt |
| Add batch normalization while using KFC-pre (ablation) | CIFAR-10 convolutional network · CIFAR-10 | training and test error optimization speed | Helped |
| Use large mini-batches of size 4096 instead of the small-mini-batch settings (ablation) | SGD and KFC-pre · CIFAR-10 and SVHN | test error before overfitting | Hurt |
Closest papers by meaning
- SKFAC: Training Neural Networks With Faster Kronecker-Factored Approximate Curvature (CVPR 2021 · similarity 0.93)
- A Trace-restricted Kronecker-Factored Approximation to Natural Gradient (AAAI 2021 · similarity 0.92)
- Kronecker-factored Curvature Approximations for Recurrent Neural Networks (ICLR 2018 · similarity 0.92)
- Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.89)
- An Improved Empirical Fisher Approximation for Natural Gradient Descent (NeurIPS 2024 · similarity 0.86)
- A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.