Research map / Optimization / Optimizers
Understanding Stochastic Natural Gradient Variational Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves the first O(1/T) non-asymptotic convergence rate for stochastic natural gradient variational inference with conjugate likelihoods, and shows that with non-conjugate likelihoods the canonical NGVI implicitly optimizes a non-convex objective.
Problem
Non-asymptotic convergence rate of stochastic natural gradient variational inference (NGVI) is unknown, even for conjugate likelihoods; behavior with non-conjugate likelihoods is also unclear.
Approach
Casts NGVI as mirror descent with the log-partition conjugate A* as distance-generating function, shows the negative ELBO is relatively 1-smooth and 1-strongly convex for conjugate likelihoods, and applies stochastic mirror descent analysis to get an O(1/T) rate with step size 2/(2+t) that is independent of condition number and initialization distance.
Key results
- Stochastic NGVI with step size 2/(2+t) achieves E[ℓ(ω̄)] − min ℓ ≤ V/(T+2) for conjugate likelihoods, matching the O(1/T) rate of SGD with no dependence on condition number or initialization distance.
- On Bayesian linear regression (Bike), KL to optimum for stochastic NGD and SGD decreases at the same O(1/T) slope, with NGD converging much faster in early iterations.
- For non-conjugate likelihoods, explicit examples (Bayesian logistic regression and Poisson regression) show the negative ELBO in expectation parameters is non-convex even for log-concave likelihoods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use stochastic NGVI instead of SGD | Bayesian linear regression · Bike | early-stage convergence speed | Helped |
| use stochastic NGVI instead of SGD | Bayesian linear regression · Bike | asymptotic optimality-gap convergence rate | No clear effect |
| use stochastic NGVI instead of SGD | Bayesian logistic regression · Mushroom | convergence speed | Helped |
| use stochastic NGVI instead of SGD | Bayesian logistic regression · MNIST subset of 1 and 7 | convergence speed | Helped |
| use the Price stochastic gradient instead of the reparameterization-trick gradient | stochastic NGVI · Bayesian logistic regression on Mushroom and MNIST | convergence | Helped |
| use step size 0.1 | stochastic NGVI · Non-conjugate likelihoods | convergence smoothness | Mixed |
| reduce the step size from 0.1 to 0.01 | stochastic NGVI · Non-conjugate likelihoods | convergence smoothness | Helped |
| use the Price gradient estimator instead of the reparameterization-trick estimator | stochastic NGVI · Quadratic log likelihood | variance of the covariance-gradient estimate: zero variance | Helped |
| use the canonical expectation-parameter NGVI update with a non-conjugate likelihood | stochastic NGVI · Non-conjugate likelihoods | convexity of the negative ELBO | Hurt |
Surprising findings
- Automatic differentiation stochastic gradients for the expectation parameter need not be negative definite, which can break valid NGD updates.
- The negative ELBO in expectation parameters is non-convex for non-conjugate log-concave likelihoods, although it has a unique minimizer.
Limitations
- Convergence rate proven only for conjugate likelihoods; non-conjugate global O(1/T) rate remains open.
- Empirical constant-factor advantage over SGD is argued heuristically rather than proven.
- Gradient variance bound given explicitly only for Bayesian linear regression with data subsampling.
Experimental setup
Datasets: Bike (UCI) Mushroom (UCI) MNIST
Benchmarks: Bike Mushroom MNIST
Metrics: KL divergence to optimal posterior negative log predictive density
Baselines: stochastic gradient descent (black-box VI)
Builds on: mirror descent stochastic mirror descent for relatively smooth functions black-box variational inference stochastic variational Gaussian processes
Closest papers by meaning
- Natural Gradient VI: Guarantees for Non-Conjugate Models (NeurIPS 2025 · similarity 0.97)
- Marginalized Stochastic Natural Gradients for Black-Box Variational Inference (ICML 2021 · similarity 0.91)
- Least squares variational inference (NeurIPS 2025 · similarity 0.90)
- Globally Convergent Variational Inference (NeurIPS 2024 · similarity 0.88)
- EigenVI: score-based variational inference with orthogonal function expansions (NeurIPS 2024 · similarity 0.88)
- Bayesian Online Natural Gradient (BONG) (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.