Research map / Optimization / Optimizers
Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdamQLR, which wraps Adam's update direction with K-FAC's Levenberg-Marquardt damping and quadratic-model learning rate selection, and studies whether these K-FAC heuristics help first-order optimisation.
Problem
Second-order optimisers for deep learning depend on stabilising heuristics (e.g. Levenberg-Marquardt damping), and it is unclear how much these heuristics, rather than the curvature model, drive second-order performance; the paper asks whether they also benefit first-order methods like Adam.
Approach
Apply K-FAC's damping adaptation (reduction-ratio rule) and quadratic-model learning rate selection, using the exact Fisher via Jacobian-vector products, on top of Adam's update direction, yielding an optimiser-agnostic wrapper (AdamQLR) with a largely untuned default setting.
Key results
- On UCI Protein, AdamQLR (Tuned) reaches test loss 0.23733 vs 0.2475 for tuned Adam, while K-FAC reaches 0.23018.
- On Fashion-MNIST, AdamQLR (Untuned) reaches test accuracy 0.87100 vs 0.87437 for Adam (Untuned) and 0.87320 for tuned Adam, with fewer total seconds (10.66 vs 26.29 for Adam).
- On UCI Energy, AdamQLR (Untuned) diverged, with test loss on the order of 1e29, while tuned AdamQLR roughly matched tuned Adam (test loss 0.000873 vs 0.000768).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive damping and quadratic-model learning-rate selection to Adam | Adam on UCI Energy · UCI Energy | test loss: 0.000768 ± \num{0.000029} -> 0.000873 ± \num{0.000021} | Hurt |
| Add adaptive damping and quadratic-model learning-rate selection to Adam | Adam on UCI Protein · UCI Protein | test loss: 0.2475 ± \num{0.0011} -> 0.23733 ± \num{0.00076} | Helped |
| Add adaptive damping and quadratic-model learning-rate selection to Adam | Adam on Fashion-MNIST · Fashion-MNIST | test loss: 0.3822 ± \num{0.0015} -> 0.37100 ± \num{0.00078} | Helped |
| Use the default untuned AdamQLR configuration instead of tuned AdamQLR | AdamQLR on Fashion-MNIST · Fashion-MNIST | test loss: 0.37100 ± \num{0.00078} -> 0.37028 ± \num{0.00060} | No clear effect |
| Add adaptive damping and quadratic-model learning-rate selection to Adam | Adam on SVHN · SVHN | test accuracy | Mixed |
| Use AdamQLR instead of Adam | Adam on CIFAR-10 · CIFAR-10 | test accuracy | Mixed |
| Use AdamQLR with untuned default hyperparameters instead of tuned AdamQLR | AdamQLR on UCI Energy · UCI Energy | training loss: 0.0003330 ± \num{0.0000087} -> \num{1.6e29} ± \num{5.3e28} | Hurt |
| Remove damping from AdamQLR (ablation) | AdamQLR on Fashion-MNIST · Fashion-MNIST | test loss | Hurt |
| Vary the initial damping value (ablation) | AdamQLR on Fashion-MNIST · Fashion-MNIST | test loss | No clear effect |
| Increase batch size (ablation) | AdamQLR on Fashion-MNIST · Fashion-MNIST | test loss | Helped |
| Rescale the selected learning rate by a constant factor (ablation) | AdamQLR on Fashion-MNIST · Fashion-MNIST | test loss | No clear effect |
Surprising findings
- AdamQLR (Untuned) diverged rapidly on UCI Energy, while K-FAC performed fine on the same task.
- Adding K-FAC's heuristics to Adam sometimes improved generalisation, but full K-FAC overfit more on Fashion-MNIST and SVHN.
Limitations
- Adam and AdamQLR may struggle on problems where Adam is known to underperform, such as CIFAR-10 training dynamics and transformers (Penn Treebank untuned variant underperforms).
- The learning-rate selection assumes an approximately convex, quadratic local surface; convergence is argued only heuristically without formal proof.
- Results show high variance and mixed benefit of heuristics, with divergence in some settings (e.g. UCI Energy untuned).
Experimental setup
Datasets: UCI Energy UCI Protein Fashion-MNIST SVHN CIFAR-10 Penn Treebank Rosenbrock function
Benchmarks: UCI Energy UCI Protein Fashion-MNIST SVHN CIFAR-10 Penn Treebank Rosenbrock function
Metrics: training loss test loss test accuracy generalisation gap wall-clock runtime
Baselines: SGD Adam K-FAC
Builds on: Adam K-FAC ASHA KFAC-JAX
Closest papers by meaning
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.89)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.87)
- AdaFisher: Adaptive Second Order Optimization via Fisher Information (ICLR 2025 · similarity 0.86)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.86)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.86)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.