Research map / Optimization / Implicit bias and generalization
Neglected Hessian component explains mysteries in sharpness regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomposes the loss Hessian into a Gauss-Newton part (exploitation) and a Nonlinear Modeling Error (NME) part (exploration), and shows via gradient penalty, weight noise and activation-function ablations that the NME explains differing generalization effects.
Problem
Why do seemingly similar second-order regularizers (SAM, weight noise, gradient penalties) have inconsistent generalization benefits, and why is the non-PSD NME part of the Hessian usually neglected?
Approach
Interprets the Gauss-Newton term as exploiting linear (NTK) structure and the Nonlinear Modeling Error matrix as exploring switches between multilinear regions; shows the NME depends on activation second derivatives, so gradient penalties are sensitive to activation choice while SAM is not, and that weight noise's equivalence to GN penalties breaks because of the NME.
Key results
- Gauss-Newton trace penalty improves ImageNet ResNet-50 accuracy by 1.6% while weight noise and Hessian trace penalty give little improvement at large sigma^2.
- With GELU, gradient penalty accuracy increases with rho up to 0.1, while with ReLU it decreases over a similar range.
- Augmented ReLU (with a Gaussian-smoothed second derivative) performs better than plain ReLU with gradient penalty and nearly matches GELU; diminished GELU performs similar to ReLU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a gradient penalty with GELU activation | ResNet-50 · ImageNet | test accuracy | Helped |
| Add a gradient penalty with ReLU activation | ResNet-50 · ImageNet | test accuracy | Hurt |
| Increase β in β-GELU while training with a gradient penalty | ResNet-50 · ImageNet and CIFAR-10 | test accuracy | Hurt |
| Increase β in β-GELU while training with SGD | ResNet-50 · ImageNet and CIFAR-10 | test accuracy | No clear effect |
| Add second-derivative information to ReLU (augmented ReLU) (ablation) | gradient-penalty training · ImageNet | test accuracy | Helped |
| Remove second-derivative information from GELU (diminished GELU) (ablation) | gradient-penalty training · ImageNet | test accuracy | Hurt |
| Use a Gauss-Newton Trace penalty instead of weight noise or a Hessian Trace penalty (ablation) | ResNet-50 · ImageNet | test accuracy: 1.6% | Helped |
| Include the NME contribution in the Hessian Trace penalty rather than exclude it with the Gauss-Newton Trace penalty (ablation) | ResNet-50 · ImageNet and CIFAR-10 | test accuracy | Hurt |
| Use SAM instead of a gradient penalty | ImageNet models with ReLU and GELU · ImageNet | test accuracy | No clear effect |
Surprising findings
- Weight noise, commonly analyzed as a penalty on the Gauss-Newton part, behaves differently from the GN trace penalty because the NME term is included, and regularizing the NME directly is detrimental.
- Penalty SAM (gradient penalty) is sensitive to activation function choice while SAM itself is insensitive to it.
Limitations
- Activation-function choice is shown to be sufficient for effects only in the studied settings; the paper does not claim it is sufficient for gradient penalties to work with larger rho.
- Analysis of NME relies on large-width/NTK approximations and gradient-flow limits.
Experimental setup
Datasets: ImageNet CIFAR-10
Benchmarks: ImageNet CIFAR-10
Metrics: test accuracy
Baselines: SAM weight noise gradient penalty Gauss-Newton trace penalty Hessian trace penalty
Builds on: Sharpness Aware Minimization Gauss-Newton matrix neural tangent kernel Hutchinson trace estimator
Closest papers by meaning
- Investigating the Overlooked Hessian Structure: From CNNs to LLMs (ICML 2025 · similarity 0.88)
- Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization (NeurIPS 2024 · similarity 0.88)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.88)
- Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise (NeurIPS 2025 · similarity 0.87)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.86)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.