Research map / Probabilistic and scientific modeling / Variational and generative sampling
Least squares variational inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LSVI, a gradient-free Monte Carlo fixed-point scheme for exponential-family variational inference that reduces each step to ordinary least squares on tempered log-target evaluations, with convergence rates and efficient Gaussian variants.
Problem
Variational inference with exponential-family approximations when gradients of the log-target are unavailable or the reparameterisation trick is impossible (discrete targets, non-differentiable or likelihood-free models); natural gradient methods need costly Fisher inversions.
Approach
The optimal exponential-family approximation solves a fixed-point equation that equals an ordinary least squares regression of log-target on sufficient statistics under the current approximation; a Monte Carlo, damped version of this iteration is shown equivalent to biased stochastic natural gradient / mirror descent, and for Gaussians the regression is reparameterised via Hermite-like statistics
Key results
- On Pima (full-covariance, d=9), LSVI converges essentially in one step and LSVI-FC in under 100 steps, performing comparably to NGD and GMMVI but less noisy.
- On MNIST (mean-field, d=784), the LSVI-MF variant runs in about 10-19 seconds (JAX, 500 iterations) versus about 25 seconds for mean-field NGD with the same sample count.
- For the variable selection task (d=92), LSVI gives a reasonable approximation of SMC posterior inclusion probabilities at about 61 seconds versus 291 seconds for SMC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LSVI-FC rather than MCMC | Bayesian synthetic likelihood for toad displacement · BSL toad displacement | runtime (seconds): $72.9$ ($\pm 2.8$) vs $268.1$ ($\pm 3.4$) | Helped |
| use LSVI rather than SMC | variable selection · Concrete Compressive Strength variable selection | runtime (seconds): $60.8$ ($\pm 0.3$) vs $290.7$ ($\pm 1.7$) | Helped |
| use LSVI-MF rather than NGD-MF | Census logistic regression with subsampling · Census logistic regression, subsampling | runtime (seconds): $12.9$ ($\pm 0.2$) vs $70.0$ ($\pm 1.7$) | Helped |
Surprising findings
- Ordinary least squares on tempered log-target evaluations converges in essentially one step on Pima, faster than gradient-based NGD.
Limitations
- Restricted to exponential variational families; mixtures are not covered.
- Convergence guarantees rely on smoothness, relative strong convexity and bounded-moment assumptions, conditioned on high-probability events.
- Gaussian approximations may be poor when the posterior is strongly non-Gaussian.
Experimental setup
Datasets: Pima Sonar Census-Income MNIST (digits 0 vs 8) Concrete Compressive Strength toad displacement (synthetic)
Benchmarks: Pima logistic regression Sonar logistic regression MNIST logistic regression Census-Income logistic regression Concrete variable selection toad BSL
Metrics: KL divergence to posterior (up to constant) runtime memory usage misclassification rate posterior inclusion probabilities
Baselines: ADVI NGD GMMVI SMC sampler MCMC (random walk)
Builds on: natural gradient variational inference mirror descent GMMVI ADVI
Closest papers by meaning
- EigenVI: score-based variational inference with orthogonal function expansions (NeurIPS 2024 · similarity 0.91)
- Natural Gradient VI: Guarantees for Non-Conjugate Models (NeurIPS 2025 · similarity 0.90)
- Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.90)
- Understanding Stochastic Natural Gradient Variational Inference (ICML 2024 · similarity 0.90)
- Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians (ICML 2024 · similarity 0.89)
- Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inference (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.