Research map / Optimization / Optimizers
AdaFisher: Adaptive Second Order Optimization via Fisher Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaFisher, an Adam-style adaptive second-order optimizer that replaces the second moment with a diagonal block-Kronecker approximation of the Fisher information matrix, and evaluates it on CIFAR, ImageNet-1k, and small language models.
Problem
First-order optimizers like Adam generalize poorly and need heavy hyper-parameter tuning, while second-order methods (K-FAC, AdaHessian, Shampoo) are computationally expensive and hard to tune at scale.
Approach
Empirically finds Kronecker factors of the FIM are diagonally concentrated, and approximates them by their diagonals (min-max normalized) combined via a Kronecker product plus Tikhonov damping; this preconditioner replaces Adam's second moment, and the square root is dropped.
Key results
- On CIFAR-10 with ResNet18 (batch 256, 200 epochs), AdaFisher reaches 96.25% vs 94.85% for Adam and 95.64% for SGD.
- On CIFAR-100 with MobileNetV3, AdaFisher reaches 77.56% vs 70.62% for Adam.
- On WikiText-2 language modeling with a scaled-down GPT-1, AdaFisherW achieves test PPL 152.72 vs 175.06 for AdamW.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Adam-style diagonal second-moment preconditioning with AdaFisher's diagonal block-Kronecker FIM preconditioning | ResNet18 · CIFAR-10 | accuracy: 95.64_{0.1} -> 96.25_{ 0.2} | Helped |
| Replace Adam-style diagonal second-moment preconditioning with AdaFisher's diagonal block-Kronecker FIM preconditioning | ResNet50 · CIFAR-100 | accuracy: 78.07_{0.4} -> 79.77_{0.4} | Helped |
| Use AdaFisher preconditioning instead of the strongest listed baseline | Tiny Swin · CIFAR-100 | accuracy: 60.21_{0.4} -> 66.05_{0.5} | Helped |
| Use AdaFisher preconditioning instead of the strongest listed baseline | FocalNet · CIFAR-100 | accuracy: 52.71_{0.5} -> 53.69_{0.3} | Helped |
| Use AdaFisher preconditioning instead of AdamW | scaled-down GPT-1 · WikiText-2 test | perplexity: 175.06 -> 152.72 | Helped |
| Use AdaFisher preconditioning instead of AdamW | scaled-down GPT-1 · PTB test | perplexity: 44.70 -> 41.15 | Helped |
| Use AdaFisher rather than other listed optimizers | CNN architectures · CIFAR-10 and CIFAR-100 | accuracy | Helped |
| Increase batch size | AdaFisher with ResNet50 · CIFAR-100 | test accuracy | Mixed |
| Vary learning rate | AdaFisher with ResNet50 · CIFAR-100 | test accuracy | No clear effect |
Surprising findings
- K-FAC and Shampoo perform very poorly on ViT-style models (e.g., K-FAC 33.08% on CIFAR10 with CCT), and Shampoo failed to converge on language modeling despite tuning.
- AdaHessian on MobileNetV3 CIFAR-100 drops to 56.58% with high variance, indicating instability.
Limitations
- Optimal only for negative log-likelihood losses in exponential-family settings where FIM equals the Generalized Gauss-Newton matrix.
- Evaluated on a limited set of models and domains; generative modeling and graph networks left to future work.
- Language modeling experiments use a scaled-down GPT-1 and a small set of benchmarks.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet ImageNet-1k WikiText-2 Penn Treebank
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k WikiText-2 Penn Treebank
Metrics: test accuracy perplexity wall-clock time accuracy mean and std over five runs
Baselines: SGD Adam AdamW K-FAC AdaHessian Shampoo LAMB LARS
Builds on: Adam K-FAC AdaHessian AdamW Shampoo
Closest papers by meaning
- Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lens (ICML 2024 · similarity 0.86)
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.86)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.86)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.86)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.86)
- ASGO: Adaptive Structured Gradient Optimization (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.