Research map / Probabilistic and scientific modeling / Variational and generative sampling
Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies variational inference over equal-weight, fixed-covariance Gaussian mixtures by viewing it as minimizing a mollified relative entropy over particle positions, deriving a descent lemma for Wasserstein gradient descent and a 1/n-type (log n / n) approximation bound in reverse KL.
Problem
Theoretical guarantees (optimization and approximation error) for variational inference when the variational family is a mixture of Gaussians, which has received little theoretical study beyond the Gaussian case.
Approach
VI over fixed-variance, equally weighted Gaussian mixtures is cast as minimizing a mollified relative entropy (KL between a Gaussian-smoothed Dirac mixture and the target), whose particle gradient flow is smooth enough to yield a descent lemma; the reverse-KL approximation error is bounded by C^2 (log n + 1)/n via an induction argument.
Key results
- Descent lemma: F_eps(mu_{l+1}) - F_eps(mu_l) <= -gamma(1 - gamma M/2) ||grad F'_eps(mu_l)||^2, implying the average squared gradient norm decays as O(1/L).
- Approximation bound: min over n-component mixtures of KL(k_eps * mu_n | mu*) <= C^2 (log(n)+1)/n, with C^2 a target-dependent constant.
- Numerical experiments with a 100-component Gaussian mixture target show the reverse KL decreasing roughly linearly in n across dimensions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| optimize the mollified KL by Wasserstein gradient descent with a sufficiently small step size | Fixed-variance Gaussian mixture VI · Wasserstein gradient descent | mollified KL: \mathcal{F}_{\epsilon}(\mu_{l+1})-\mathcal{F}_{\epsilon}(\mu_l)\leq -\gamma \left(1-\frac{\gamma}{2}M \right) \|\nabla \ | Helped |
| increase the number of equally weighted mixture components | Fixed-variance Gaussian mixture VI · Gaussian mixture approximation | reverse KL: C_{\mu^{\star}}^2 \frac{\log(n) + 1}{n} | Helped |
| allow non-fixed mixture weights (ablation) | Fixed-variance Gaussian mixture VI · Gaussian mixture approximation | reverse KL rate: 1/n | Helped |
| mollify the KL objective with a positive ε | Variational inference objective · Wasserstein gradient descent | objective smoothness | Helped |
| mollify the KL objective | Variational inference objective · Wasserstein geometry | displacement convexity | Hurt |
| take ε to zero | Mollified KL objective · Wasserstein geometry | Wasserstein Hessian: \Hess_{\mu}\mathcal{F}_{\epsilon}(\psi,\psi) \xrightarrow[\varepsilon \to 0]{} \Hess_{\mu}\KL(\psi,\psi) | Helped |
Limitations
- Mixture weights and covariances are fixed; extending to optimized weights (Fisher-Rao dynamics) or variable covariances is left open.
- The descent lemma relies on an empirically verified bounded second-moment assumption along iterates rather than a proof.
- Mollified KL loses convexity; non-asymptotic Hessian bounds in epsilon are not provided.
Experimental setup
Datasets: synthetic Gaussian mixture target (100 components)
Metrics: mollified KL objective decrease average squared Wasserstein gradient norm reverse KL to target
Baselines: Katsevich & Rigollet (2023) forward KL rate Gaussian VI
Builds on: Wasserstein gradient flows Bures-Wasserstein gradient descent Katsevich & Rigollet forward-KL mixture approximation
Closest papers by meaning
- Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.95)
- Variational inference via Wasserstein gradient flows (NeurIPS 2022 · similarity 0.92)
- Generalized Variational Inference via Optimal Transport (AAAI 2024 · similarity 0.90)
- Particle Semi-Implicit Variational Inference (NeurIPS 2024 · similarity 0.89)
- Least squares variational inference (NeurIPS 2025 · similarity 0.89)
- Symmetry Guarantees Statistic Recovery in Variational Inference (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.