Research map / Optimization / Optimizers
Global curvature for second-order optimization of neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives the exact structure of the global (parameter-averaged) gradient covariance of MLPs from permutation/sign/orthogonal symmetries, and uses its O(d^2+L^2)-factor Kronecker structure to build a cheap second-order optimizer (SymO) that converges faster than GD, Adam, KFAC and Shampoo on synthetic data.
Problem
Second-order optimizers need to construct and invert curvature matrices with O(p^2) elements for p parameters, which is computationally prohibitive for neural networks.
Approach
Use discrete and orthogonal symmetries of MLPs together with invariance of the initialization distribution to show that the global (ensemble-averaged) gradient covariance has a Kronecker-factored structure determined by a small number of factors; estimate these factors from a single model and compute the inverse square root via a surrogate matrix of small width.
Key results
- On a two-layer MLP (widths 100-70-40, 9800 parameters) with linear and Tanh activations, SymO converges faster than GD, Adam, KFAC and Shampoo in 100 full-batch iterations, with nearly identical time per iteration.
- Theory predicts the gradient covariance structure for two- and three-layer Tanh MLPs and three-layer ReLU MLPs, matching empirical covariance from 10000-100000 initializations.
- Single-model factor estimates correlate with ground truth at 0.67-0.97 for Phi1 when hidden width increases from 10 to 10000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SymO instead of GD, Adam, KFAC, or Shampoo | Two-layer MLP with linear activations on synthetic data · Synthetic data, linear activation | convergence speed | Helped |
| Use SymO instead of GD, Adam, KFAC, or Shampoo | Two-layer MLP with Tanh activations · Synthetic data, Tanh activation | convergence speed | Helped |
| Increase hidden-layer width when estimating curvature factors from a single model | Single-model factor estimates · MLPs with layer widths (100, 10, 100), (100, 100, 100), (100, 1000, 100), and (100, 10000, 100) | correlation with ground truth: Φ1: 0.67±0.05 -> 0.90±0.02 -> 0.96±0.01 -> 0.97±0.01; Ψ1: 0.38±0.06 -> 0.61±0.04 -> 0.64±0.03 -> 0.65±0.03; Φ2: 0.30±0.0 | Helped |
Surprising findings
- For Tanh MLPs, inter-layer covariance blocks vanish when a third layer is added, so the covariance becomes block-diagonal across layers.
- The global curvature is not inherently block-diagonal, contrary to the usual block-diagonal Kronecker approximations.
Limitations
- Experiments limited to two-layer MLPs on synthetic data with full-batch training.
- Global (ensemble-averaged) curvature is used as a proxy for local curvature without analysis of approximation error.
- Assumes invariant initialization distribution and orthogonal symmetries; Adam not equivariant under full orthogonal group.
Experimental setup
Datasets: synthetic Gaussian inputs with teacher-student targets
Benchmarks: synthetic teacher-student regression
Metrics: test loss vs iterations
Baselines: gradient descent Adam KFAC Shampoo
Builds on: KFAC Shampoo Adam Chen et al. 1993 sign-flip symmetry Kohler et al. 2019 invariant distributions
Closest papers by meaning
- Deconstructing the Goldilocks Zone of Neural Network Initialization (ICML 2024 · similarity 0.89)
- Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.87)
- Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms (NeurIPS 2024 · similarity 0.87)
- FOSI: Hybrid First and Second Order Optimization (ICLR 2024 · similarity 0.87)
- Compositional Curvature Bounds for Deep Neural Networks (ICML 2024 · similarity 0.87)
- GradMetaNet: An Equivariant Architecture for Learning on Gradients (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.