Research map / Optimization / Optimizers
Natural Gradient VI: Guarantees for Non-Conjugate Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives sufficient conditions for relative smoothness of the non-conjugate mean-field Gaussian variational loss w.r.t. the log-partition conjugate, proposes a projected stochastic NGVI (Proj-SNGD) with non-Euclidean projection, and proves O(1/sqrt T) stationarity and O(1/T) global convergence under log-concave likelihoods.
Problem
Convergence guarantees for natural gradient variational inference (NGVI) in non-conjugate models, where the variational loss is non-convex and not globally relatively smooth.
Approach
Shows that for mean-field Gaussian VI the negative ELBO is relatively smooth w.r.t. the log-partition conjugate A* on compact parameter sets (with polynomial constants), uses a projected SMD-style NGVI update with clipping onto that set, and exploits hidden convexity under the Cholesky reparameterization to obtain fast global rates under log-concave likelihoods.
Key results
- Proj-SNGD converges faster and is stable on Poisson regression where SNGD with sigma_0^2=2 shows instability at the initial iteration, with projection fixing the increase in objective.
- On MNIST logistic regression, non-Euclidean algorithms (Proj-SNGD, SNGD) reach the threshold objective of 700 in roughly 1000-2000 iterations for step-size gamma_0 in [0.05, 0.2], while Euclidean methods only do so at gamma_0=0.02.
- Theory: O(1/sqrt T) rate to a stationary point in the stochastic case and O(1/T) global rate (linear in the noiseless case) under concave log-likelihood with an additional boundary assumption.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add non-Euclidean projection | SNGD (Proj-SNGD) · Poisson regression | numerical stability | Helped |
| add non-Euclidean projection | SNGD (Proj-SNGD) · Poisson regression | convergence speed | Helped |
| use non-Euclidean rather than Euclidean optimization | logistic regression · MNIST (labels 6 or 8) | convergence speed | Helped |
| use non-Euclidean rather than Euclidean optimization | logistic regression · MNIST (labels 6 or 8), objective ≤ 700 | iterations to threshold: around 1000–2000 iterations; Euclidean algorithms only do so at γ₀=0.02 | Helped |
| add non-Euclidean projection | SNGD (Proj-SNGD) · MNIST (labels 6 or 8) | objective | Helped |
Surprising findings
- Relative smoothness constant for the non-conjugate log-likelihood term grows exponentially with the initial variance sigma_0^2 in the univariate Poisson example, so unprojected SNGD can become unstable.
- Logistic regression objective is shown to be non-convex in the variance parameter for small variances, so the 1-smoothness of conjugate models fails.
Limitations
- Landscape results (relative smoothness, PL) hold only on a compact parameter set; global smoothness remains open.
- Analysis restricted to mean-field Gaussian family; full-covariance not covered.
- The O(1/T) rate requires an additional boundary assumption on the descent direction that may be hard to verify.
Experimental setup
Datasets: MNIST subset (labels 6 and 8, n=11769, d=784) synthetic Poisson regression data (x=0.9, y=24)
Benchmarks: MNIST logistic regression subset Poisson regression toy problem
Metrics: objective function value gap iterations to reach threshold objective
Baselines: SNGD Prox-SGD Proj-SGD
Builds on: stochastic natural gradient variational inference stochastic mirror descent relative smoothness Wu and Gardner SNGD analysis hidden convexity
Closest papers by meaning
- Understanding Stochastic Natural Gradient Variational Inference (ICML 2024 · similarity 0.97)
- Least squares variational inference (NeurIPS 2025 · similarity 0.90)
- Marginalized Stochastic Natural Gradients for Black-Box Variational Inference (ICML 2021 · similarity 0.90)
- EigenVI: score-based variational inference with orthogonal function expansions (NeurIPS 2024 · similarity 0.90)
- Revisiting Unbiased Implicit Variational Inference (ICML 2025 · similarity 0.88)
- Bayesian Online Natural Gradient (BONG) (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.