Research map / Optimization / Implicit bias and generalization
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add batch normalization (ablation) | non-BN networks · CIFAR-10 and ImageNet training | Hessian outlier eigenvalues | Helped |
| remove batch normalization (ablation) | BN networks · CIFAR-10 and ImageNet training | Hessian outlier eigenvalues | Hurt |
| compute batch-normalization statistics from the full population instead of mini-batches (ablation) | Resnet-32 · CIFAR-10 training | optimization speed | Hurt |
| compute batch-normalization statistics from the full population instead of mini-batches (ablation) | Resnet-32 · CIFAR-10 training | Hessian outlier eigenvalues | Hurt |
| reduce the learning rate (ablation) | Resnet-32 · CIFAR-10 training, step 40000 | Hessian eigenvalue magnitude | Helped |
| remove residual connections (ablation) | Resnet-32 · CIFAR-10, step 40000 | Hessian eigenvalue magnitude | Helped |
| remove data augmentation (ablation) | Resnet-32 · CIFAR-10 validation | accuracy: 92% -> around 85% | Hurt |
| reduce final fully connected layer width from 2048 to 256 neurons (ablation) | VGG-11 · CIFAR-10 validation | accuracy: 83% -> 81% | Hurt |
| use a degree 90 Chebyshev polynomial approximation instead of stochastic Lanczos quadrature (ablation) | Hessian spectrum estimation · MNIST feed-forward network | spectral-density estimation quality | Hurt |
Closest papers by meaning
- The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network (NeurIPS 2018 · similarity 0.90)
- A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization (AAAI 2021 · similarity 0.89)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.88)
- Investigating the Overlooked Hessian Structure: From CNNs to LLMs (ICML 2025 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- Spectral Preconditioning for Gradient Methods on Graded Non-convex Functions (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.