Research map / Optimization / Implicit bias and generalization
Variational Learning Finds Flatter Solutions at the Edge of Stability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes implicit regularization of variational learning through the edge-of-stability lens, showing that posterior variance and number of Monte-Carlo samples lower the stability threshold below 2/ρ, so VL finds flatter solutions than GD.
Problem
Understanding the implicit regularization of variational learning (VL) and why it finds flatter solutions than gradient descent, which existing PAC-Bayes/MDL/marginal likelihood theories do not explain.
Approach
Derives an exact sufficient stability threshold for variational gradient descent (VGD) on quadratics, showing it is 2/ρ times a Variational Factor depending on posterior variance and sample count N_s, then extends the edge-of-stability hypothesis to deep networks, and tests it empirically.
Key results
- Theorem gives a stability threshold for variational GD of 2/ρ times a Variational Factor that is strictly below 1, so VGD is stable only for lower curvature than GD.
- On CIFAR-10, VGD with larger posterior variance and fewer Monte-Carlo samples yields lower sharpness and higher test accuracy than GD across MLP, ResNet-20 and ViT.
- Normalized sharpness of MLPs tracks the predicted Variational Factor across learning rates and posterior variances.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use isotropic Gaussian Variational GD instead of full-batch GD | CIFAR-10 MLP, ResNet-20, and ViT · CIFAR-10 | sharpness | Helped |
| use isotropic Gaussian Variational GD instead of full-batch GD | CIFAR-10 MLP, ResNet-20, and ViT · CIFAR-10 | test accuracy | Helped |
| increase isotropic Gaussian posterior variance (ablation) | Variational GD · CIFAR-10 MLP | sharpness | Helped |
| decrease the number of posterior samples (ablation) | Variational GD · CIFAR-10 MLP | sharpness | Helped |
| decrease the number of posterior samples (ablation) | Variational GD on a quadratic loss · Quadratic loss | descent probability | Hurt |
| use heavier-tailed Student-t weight perturbations (smaller α) (ablation) | Variational GD · CIFAR-10 MLP | sharpness | Helped |
| use heavier-tailed Student-t weight perturbations (smaller α) (ablation) | Variational GD · CIFAR-10 MLP | test accuracy | Helped |
| use IVON instead of Adam | MLP · Adaptive edge of stability experiment | preconditioned sharpness | Helped |
| increase learning rate (ablation) | IVON · Variational objective experiment | variational objective | Helped |
| decrease batch size (ablation) | IVON · Variational objective experiment | variational objective | Helped |
| increase posterior variance to 2 or 5 (ablation) | Variational GD · CIFAR-10 MLP, panel (c) | convergence | Hurt |
Surprising findings
- Heavy-tailed (Student-t) weight perturbations give even lower sharpness than Gaussian perturbations.
- Training did not converge for some large-variance settings in the ResNet experiment.
Limitations
- Stability threshold is derived for quadratic losses and only sufficient, not necessary, for VGD descent; extension to deep networks is empirical.
- Exact threshold for IVON/adaptive natural-gradient VL is not derived and is left to future work.
- Heavy-tailed noise lacks theoretical analysis.
Experimental setup
Datasets: CIFAR-10 ImageNet
Benchmarks: CIFAR-10
Metrics: sharpness (top Hessian eigenvalue) test accuracy descent probability
Baselines: gradient descent AdamW Adam sharpness aware minimization
Builds on: edge of stability sharpness aware minimization IVON variational online newton adaptive edge of stability
Closest papers by meaning
- Characterizing the Edge of Stability in Variational Training Without Priors (NeurIPS 2026 · similarity 0.90)
- Globally Convergent Variational Inference (NeurIPS 2024 · similarity 0.87)
- A Minimalist Example of Edge-of-Stability and Progressive Sharpening (NeurIPS 2025 · similarity 0.87)
- Variational Learning is Effective for Large Deep Networks (ICML 2024 · similarity 0.86)
- Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability (NeurIPS 2025 · similarity 0.86)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.