Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Variational Inference with Mixtures of Isotropic Gaussians
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes variational inference over uniform-weight mixtures of isotropic Gaussians, optimizing component means by gradient descent and variances by Bures-Wasserstein or entropic mirror descent updates.
Problem
Gaussian VI with full-covariance mixtures is memory- and compute-heavy in high dimension, while single Gaussians cannot capture multimodal posteriors; normalizing flows suffer mode collapse.
Approach
Restrict the mixture family to equally weighted isotropic Gaussians (one variance per component) and derive explicit updates: gradient descent on means, and either a Bures-Wasserstein (IBW) or entropic mirror descent (MD) step on each variance, which keeps variances positive and costs O(d) per component.
Key results
- Increasing the number of components (N=20) gives the lowest final KL on a 5-component 2D Gaussian mixture target, and per-component variances (IBW, MD) outperform shared-variance versions (IBW-s, MD-s).
- IBW performs comparably to full-covariance BW in KL while running faster and using memory linear rather than quadratic in dimension (N=15 experiments).
- On MNIST the mean-field version of the method reaches the same order of performance for IBW and MD, with positivity of variances guaranteed by construction unlike NGD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the mixture size to N=20 (ablation) | IBW and MD · 2D Gaussian-mixture target | KL | Helped |
| allow each component to optimize its own variance instead of sharing one variance (ablation) | IBW and MD · 2D Gaussian-mixture target | KL | Helped |
| use normalizing flows instead of the proposed methods | 2D Gaussian-mixture target · 2D Gaussian-mixture target | speed | Hurt |
| use normalizing flows instead of the proposed methods | 2D Gaussian-mixture target · 2D Gaussian-mixture target | KL | No clear effect |
| use NGD with mean updates rescaled by the variance | NGD · High-dimensional Gaussian-mixture target | convergence speed | Helped |
| use NGD with mean updates rescaled by the variance | NGD · High-dimensional Gaussian-mixture target | sensitivity to initialization | Hurt |
| use isotropic rather than full covariance matrices | IBW versus BW · High-dimensional Gaussian-mixture target | time per iteration | Helped |
| use isotropic rather than full covariance matrices | IBW versus BW · High-dimensional Gaussian-mixture target | KL | Hurt |
| use IBW or MD instead of NGD | Bayesian inference methods · MNIST BNN posterior | variance positivity | Helped |
| use IBW or MD | Bayesian inference methods · MNIST BNN posterior | performance | No clear effect |
Surprising findings
- NGD's mean update rescales gradients by the variance, which speeds convergence on spread-out components but makes it sensitive to initialization and can drive the variance negative when means are poorly initialized.
Limitations
- No formal convergence or optimization guarantees for the proposed schemes (stated as future work).
- Approximation-error comparison against full-covariance mixtures is not established theoretically.
- Relies on Monte Carlo estimates of expectations with B samples per component; MNIST required a mean-field simplification.
Experimental setup
Datasets: UCI breast_cancer UCI wine boston housing MNIST
Benchmarks: 2D Gaussian mixture target 20D Gaussian mixture target Bayesian logistic regression Bayesian neural network regression MNIST Bayesian neural network
Metrics: KL divergence test accuracy test RMSE time per iteration
Baselines: BW (full-covariance Bures-Wasserstein) GD (shared-variance mean-only gradient descent) NGD (natural gradient descent for mixtures) Real NVP normalizing flow HMC ADVI Diagonal and K-FAC Laplace approximations
Builds on: Bures-Wasserstein gradient descent on Gaussians JKO scheme entropic mirror descent natural gradient VI for Gaussian mixtures Wasserstein gradient flow for Gaussian mixtures
Closest papers by meaning
- Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians (ICML 2024 · similarity 0.95)
- Generalized Variational Inference via Optimal Transport (AAAI 2024 · similarity 0.90)
- Least squares variational inference (NeurIPS 2025 · similarity 0.90)
- EigenVI: score-based variational inference with orthogonal function expansions (NeurIPS 2024 · similarity 0.90)
- Efficient Mixture Learning in Black-Box Variational Inference (ICML 2024 · similarity 0.89)
- SoftCVI: Contrastive variational inference with self-generated soft labels (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.