Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Variational Learning is Effective for Large Deep Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that IVON, an Adam-like variational optimizer using a reparameterization-trick Hessian estimate, can train large networks (GPT-2 up to 773M, ResNet-50) from scratch at near-Adam cost while improving accuracy and uncertainty.
Problem
Variational learning is widely believed to be ineffective for large deep networks, with high cost and poor accuracy, so no variational method had been shown to train LLMs from scratch at Adam-like cost.
Approach
Adapts the Improved Variational Online Newton (Riemannian Bayesian learning rule) into an Adam-like optimizer that estimates the Hessian via a reparameterization-trick gradient-times-noise product (no per-example gradients), adds a positivity-preserving term, and removes the square root so the update is Newton-like; the posterior variance comes for free from the optimizer state.
Key results
- On GPT-2 pretraining over ~49.2B tokens, IVON lowers validation perplexity vs AdamW: 12.6 vs 13.0 (773M), 14.1 vs 14.5 (355M), 17.9 vs 18.1 (125M).
- On ImageNet ResNet-50, IVON reaches 77.46% top-1 accuracy and 0.022 ECE versus AdamW's 75.16% and 0.066 ECE.
- On CIFAR-10 ResNet-20, IVON gets 92.71% accuracy and 0.219 NLL, outperforming SWAG (92.45%) and VOGN (92.37%); Multi-IVON (94.37%) beats deep ensembles (93.57%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IVON instead of AdamW | ResNet-50 · ImageNet | Top-1 accuracy: 75.16 ± 0.14 -> 77.46 ± 0.07 | Helped |
| Use IVON instead of AdamW | ResNet-50 · ImageNet | ECE: 0.066 ± 0.002 -> 0.022 ± 0.002 | Helped |
| Use IVON instead of AdamW | GPT-2 · OpenWebText | validation perplexity: 18.1 to 17.9; 14.5 to 14.1; 13.0 to 12.6 | Helped |
| Use IVON instead of AdamW | DeBERTAv3 · GLUE | task scores: MNLI 91.3 -> 91.6; QNLI 95.7 -> 95.7; QQP 93.1 -> 93.0; RTE 91.0 -> 91.7; SST2 96.5 -> 96.9; MRPC 91.0 -> 91.9; CoLA 74. | Mixed |
| Use IVON instead of AdamW | RoBERTa · GLUE | task scores: MNLI 87.7 -> 87.8; QNLI 92.8 -> 92.6; QQP 90.9 -> 90.8; RTE 80.9 -> 80.6; SST2 94.8 -> 95.0; MRPC 85.8 -> 87.3; CoLA 63. | Mixed |
| Use the reparameterization-based Hessian estimator instead of squared-gradient estimation (ablation) | IVON · CIFAR-10, ResNet-20 | accuracy: 88.81 ± 0.31 -> 92.64 ± 0.13 | Helped |
| Use the reparameterization-based Hessian estimator instead of Gauss-Newton (ablation) | IVON · CIFAR-10, ResNet-20 | memory (MB): 645MB -> 363MB | Helped |
| Set Hessian initialization h₀ to 0.001 instead of 0.5 (ablation) | IVON · CIFAR-10, ResNet-20 | accuracy: 92.71 ± 00.07 -> 9.89 ± 00.24 | Hurt |
| Use a mixture of five independently trained IVON models instead of a single IVON model | IVON · CIFAR-10, ResNet-20 | accuracy: 92.71 ± 0.07 -> 94.37 ± 0.13 | Helped |
| Use the IVON Hessian estimate obtained during training instead of the squared-gradient estimator | masked-language model merging · IMDB, Yelp, RottenTomatoes, SST2, Amazon | Avg. score: 93.9 -> 94.0 | No clear effect |
Surprising findings
- AdamW overfits on smaller datasets (TinyImageNet, CIFAR-100) when trained 200 epochs, while IVON does not.
- IVON with a squared-gradient Hessian estimator underperforms the reparameterization-based estimator, and the Gauss-Newton estimator adds significant memory and time overhead without clear test gains.
Limitations
- Batch normalization layers do not work well with IVON.
- Using multiple MC samples increases computation cost, which the authors believe is hard to avoid.
- Transformer hyperparameters were tuned by grid search on a smaller model, so better configurations may exist; the DeBERTa AdamW reference numbers could not be reproduced.
Experimental setup
Datasets: OpenWebText ImageNet CIFAR-10 CIFAR-100 TinyImageNet GLUE IMDB Yelp Amazon RottenTomatoes SST2
Benchmarks: OpenWebText validation perplexity ImageNet CIFAR-10 CIFAR-100 TinyImageNet SVHN OOD Flowers102 OOD GLUE model merging sentiment datasets
Metrics: accuracy validation perplexity NLL ECE Brier score AUROC FPR@95 runtime memory
Baselines: AdamW SGD AdaHessian BBB Laplace MC-Dropout SWAG VOGN Deep Ensembles
Builds on: Variational Online Newton (VON) Improved Variational Online Newton Bayesian Learning Rule VOGN AdamW
Closest papers by meaning
- Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (ICML 2018 · similarity 0.87)
- Variational Learning Finds Flatter Solutions at the Edge of Stability (NeurIPS 2025 · similarity 0.86)
- MARS: Unleashing the Power of Variance Reduction for Training Large Models (ICML 2025 · similarity 0.85)
- Gradient-Variation Online Learning under Generalized Smoothness (NeurIPS 2024 · similarity 0.85)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.85)
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.