Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
ELBOing Stein: Variational Bayes with Stein Mixture Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stein Mixture Inference (SMI), which uses each SVGD-style particle to parameterize a guide component in a uniform mixture and optimizes a mixture ELBO via Nonlinear SVGD, avoiding variance collapse with fewer particles.
Problem
SVGD underestimates posterior variance (variance collapse), so overcoming it requires many particles as model dimension grows, which is infeasible even for small Bayesian neural networks.
Approach
Each particle parameterizes a variational guide in a uniform mixture; the mixture ELBO plus an entropy term is optimized with Nonlinear SVGD. With alpha=1 this is a proper ELBO, and it reduces to SVGD (point-mass guides) and OVI (one particle) as special cases.
Key results
- On the standard UCI splits, SMI achieves the best NLL on most datasets while matching MAP on RMSE, and is the only VI method that does not collapse in the Gaussian variance experiment.
- On MNIST with a 1-layer BNN, SMI reaches accuracy 0.957 and NLL 0.039, outperforming SVGD (0.949, 0.054) and ASVGD (0.949, 0.053).
- In the recovery-point experiment, SVGD needs 8 particles to match 5-particle SMI on the low-dimensional BNN and more than 256 on the moderate-dimensional BNN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SMI instead of MAP | 1-layer BNN · MNIST | NLL: 0.050 ± 0.002 -> 0.039 ± 0.003 | Helped |
| Use SMI instead of OVI | 1-layer BNN · MNIST | ECE: 0.085 ± 0.010 -> 0.148 ± 0.012 | Hurt |
| Use SMI instead of MAP | 2-layer BNN · MNIST | NLL: 0.044 ± 0.003 -> 0.042 ± 0.005 | Helped |
| Use SMI instead of MAP | 2-layer BNN · MNIST | Brier score: 0.066 ± 0.000 -> 0.067 ± 0.003 | Hurt |
| Use SMI instead of OVI | 2-layer BNN · MNIST | ECE: 0.084 ± 0.009 -> 0.150 ± 0.014 | Hurt |
| Use SMI instead of SVGD or ASVGD | moderate BNN · 1D regression, wave dataset, Between and Entire regions | median recovery point R: >256 | Helped |
| Use one SMI particle with a Gaussian guide instead of 20 SMI particles | SMI · standard multivariate Gaussian variance estimation | dimension marginal variance | Helped |
| Reduce α to at most 1 | SMI with 20 particles · standard multivariate Gaussian variance estimation | dimension marginal variance | Helped |
Surprising findings
- NUTS outperforms SMI on NLL in the UCI benchmark, and SMI is only the best among variational methods.
- A single-particle SMI with a Gaussian guide recovers the multivariate Gaussian exactly regardless of dimension.
Limitations
- Variational approximation can be misspecified with too many particles or a poor parametric guide family.
- Choice of kernel is hard to determine and only RBF kernel is studied.
- Evaluated only on small to moderate-sized models; not tested on high-dimensional models.
Experimental setup
Datasets: synthetic 1D wave regression UCI regression benchmark MNIST
Benchmarks: UCI regression benchmark MNIST synthetic 1D wave regression Gaussian variance estimation
Metrics: NLL RMSE accuracy confidence Brier score ECE MCE Frobenius distance LPPD
Baselines: SVGD ASVGD OVI MAP NUTS RESVGD
Builds on: Stein variational gradient descent Nonlinear SVGD mean-field variational inference NumPyro
Closest papers by meaning
- Stein Variational Gradient Descent With Matrix-Valued Kernels (NeurIPS 2019 · similarity 0.92)
- Bayesian Posterior Approximation via Greedy Particle Optimization (AAAI 2019 · similarity 0.92)
- Approximate Bayesian Inference with Stein Functional Variational Gradient Descent (ICLR 2023 · similarity 0.92)
- Stein Variational Gradient Descent as Moment Matching (NeurIPS 2018 · similarity 0.92)
- Message Passing Stein Variational Gradient Descent (ICML 2018 · similarity 0.92)
- Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent (NeurIPS 2023 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.