Research map / Architectures and efficiency / RNNs and normalization
Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use K-FAC (either flavour) instead of first-order reference optimiser | GraphNetwork · ogbg-molpcba | steps to target mAP: 31{,}887 -> 15{,}566 | Helped |
| use K-FAC (either flavour) instead of NAdamW reference | ViT · ImageNet | steps to target accuracy: 122{,}139 -> 87{,}464 | Helped |
| K-FAC-reduce instead of K-FAC-expand | K-FAC · Wide ResNet CIFAR-10 | update step time (s): 1.36 -> 0.63 at batch 1024 | Helped |
| K-FAC-reduce instead of K-FAC-expand | K-FAC · ogbg-molpcba | steps to target mAP: 16{,}106 -> 15{,}566 | No clear effect |
| K-FAC-reduce instead of K-FAC-expand | K-FAC · ImageNet ViT | steps to target accuracy: 87{,}464 -> 92{,}550 | No clear effect |
| K-FAC-reduce instead of K-FAC-expand for weight decay selection | Wide ResNet marginal likelihood · CIFAR-10 with data augmentation | NLL: 0.24 -> 0.35 | Hurt |
| K-FAC-reduce instead of K-FAC-expand | nanoGPT GGN computation · DART | computation time: 70% of expand time | Helped |
Closest papers by meaning
- Kronecker-factored Curvature Approximations for Recurrent Neural Networks (ICLR 2018 · similarity 0.94)
- SKFAC: Training Neural Networks With Faster Kronecker-Factored Approximate Curvature (CVPR 2021 · similarity 0.93)
- Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.91)
- Global curvature for second-order optimization of neural networks (ICML 2025 · similarity 0.85)
- Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices (NeurIPS 2024 · similarity 0.85)
- Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lens (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.