Research map / Optimization / Optimizers
Inefficiency of K-FAC for Large Batch Size Training
Linjian Ma, Gabe Montague, Jiayu Ye, Zhewei Yao, Amir Gholami, Kurt Keutzer, Michael Mahoney
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase batch size | K-FAC · CIFAR-10 and SVHN image classification | iteration speedup | Hurt |
| Increase batch size | SGD · CIFAR-10 and SVHN image classification | iteration speedup | Hurt |
| Use K-FAC instead of SGD | K-FAC · CIFAR-10 and SVHN image classification | training and test performance | Hurt |
| Increase batch size under a fixed epoch budget | K-FAC · CIFAR-10, ResNet20 | hyperparameter robustness | Hurt |
| Increase batch size under a fixed iteration budget | K-FAC · CIFAR-10, ResNet20 | hyperparameter robustness | Improved |
| Increase training time | K-FAC · CIFAR-10, ResNet20 | hyperparameter robustness | Improved |
Closest papers by meaning
- Distributed Second-Order Optimization using Kronecker-Factored Approximations (ICLR 2017 · similarity 0.90)
- Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures (NeurIPS 2023 · similarity 0.89)
- Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training (JMLR 2022 · similarity 0.88)
- Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (NeurIPS 2019 · similarity 0.87)
- How Does Critical Batch Size Scale in Pre-training? (ICLR 2025 · similarity 0.86)
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
