Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Bayesian Online Natural Gradient (BONG)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bayesian online natural gradient (BONG), a single natural-gradient step on expected log-likelihood from the prior predictive (no KL term), with deterministic linearized variants for Gaussian variational families, evaluated on MNIST and SARCOS.
Problem
Online sequential Bayesian inference for neural networks via variational Bayes is costly: iterated natural-gradient VI requires inner loops and sampling to estimate expected log-likelihood gradients and Hessians.
Approach
Drop the KL regularizer toward the previous posterior and take one unit-learning-rate natural gradient step on the expected log-likelihood starting at the prior predictive; the prior acts as implicit regularization. This recovers exact conjugate Bayesian updates, and linearizing the network yields closed-form Kalman-like updates.
Key results
- On MNIST with a CNN, BONG-LIN-HESS with DLR-10 is the clearest winner across datasets and variational families, with BLR-LIN-HESS nearly matching it but being much slower.
- BONG with a single step matches BLR (with 1 iteration) on SARCOS while BBB and BOG perform much worse.
- Linearized methods (LIN-HESS, LIN-EF) outperform MC-based variants and run faster than MC-EF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use linearized Hessian or linearized empirical Fisher instead of Monte Carlo approximations (ablation) | BONG · MNIST | NLPD | Helped |
| Use natural gradient instead of gradient descent (ablation) | BONG vs BOG · MNIST | NLPD | Helped |
| Use implicit regularization by taking one natural-gradient step from the prior instead of iterating on the VI loss with a KL term (ablation) | BONG vs BLR · MNIST | NLPD | Helped |
| Use Lin-Hess instead of Lin-EF (ablation) | BONG · MNIST | NLPD | Helped |
| Combine one-step NGD, implicit prior regularization, and linearization | BONG-Lin-Hess · Across datasets and variational families | predictive performance | Helped |
| Use 10 rather than 1 iteration per step (ablation) | BBB · SARCOS | NLPD | Helped |
| Use 10 rather than 1 iteration per step (ablation) | BLR and BBB · SARCOS | runtime | Hurt |
| Use 1 iteration per step | BONG vs BLR · SARCOS | NLPD | No clear effect |
| Use 1 iteration per step | BOG vs BBB · SARCOS | NLPD | No clear effect |
| Use Lin-Hess instead of MC-EF (ablation) | BONG · SARCOS | NLPD | Helped |
| Use DLR rank 10 instead of DLR rank 1 (ablation) | BONG · MNIST | NLPD | Helped |
| Use natural rather than moment parameterization for the diagonal Gaussian (ablation) | BONG · MNIST | NLPD | No clear effect |
Surprising findings
- Dropping the KL term and using a single natural-gradient step outperforms the iterated BLR method that retains the KL, despite being simpler and cheaper.
- Monte Carlo Hessian estimation (MC-HESS) works very poorly even with M=1000 samples.
Limitations
- Experiments use relatively small models and datasets (MNIST CNN, small SARCOS MLP).
- Conjugate-case optimality is only motivating; non-conjugate performance is empirical.
- Other methods are sensitive to learning rate, while the comparison requires tuning for them.
Experimental setup
Datasets: MNIST SARCOS
Benchmarks: MNIST SARCOS
Metrics: NLPD NLL misclassification rate expected calibration error
Baselines: BLR BBB BOG RVGA CM-EKF VD-EKF LO-FI
Builds on: Bayesian learning rule RVGA Bayes by backprop extended Kalman filter LO-FI
Closest papers by meaning
- Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm (NeurIPS 2016 · similarity 0.90)
- Variational Inference for Sequential Data with Future Likelihood Estimates (ICML 2020 · similarity 0.90)
- The Variational Predictive Natural Gradient (ICML 2019 · similarity 0.90)
- Deterministic Variational Inference for Robust Bayesian Neural Networks (ICLR 2019 · similarity 0.90)
- Online Variational Sequential Monte Carlo (ICML 2024 · similarity 0.89)
- Generalized Variational Inference via Optimal Transport (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.