Research map / Probabilistic and scientific modeling / Variational and generative sampling
Particle Semi-Implicit Variational Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Particle Variational Inference (PVI), which optimizes the mixing distribution of a semi-implicit variational family via particles from a Euclidean-Wasserstein gradient flow of the regularized ELBO, and gives existence, uniqueness, and propagation-of-chaos results for a gamma-smoothed flow.
Problem
Semi-implicit variational inference (SIVI) normally has intractable variational densities when the mixing distribution is implicit, so the ELBO cannot be optimized directly and existing methods rely on bounds, inner MCMC, or minimax objectives.
Approach
Keep the kernel parametric but optimize the mixing distribution directly over all probability measures, using a particle approximation of a Euclidean-Wasserstein gradient flow of a regularized free energy, so the ELBO is optimized without upper bounds, inner MCMC, or minimax objectives.
Key results
- On toy density estimation, PVI achieves rejection rate 0.06 / 0.05 / 0.06 on Banana / Multimodal / X-Shape, the only method with rejection rate minus std below the 0.05 nominal level on all three problems.
- On Bayesian neural network regression, PVI reports the lowest test RMSE on Concrete (0.43), Protein (0.87) and Yacht (0.13) versus UVI, SVI and SM.
- Ablation: with a Skip kernel, PVI learns a bimodal mixing distribution at mu=4 where PVIZero (fixed mixing distribution) suffers mode collapse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize the mixing distribution with PVI rather than use UVI | SIVI · Banana | sliced Wasserstein distance: 0.11_{0.03} -> 0.17_{0.01} | Hurt |
| Use PVI instead of the strongest baseline | SIVI · Multimodal | rejection rate: 0.13_{0.06} -> 0.05_{0.01} | Helped |
| Use PVI instead of the strongest baseline | SIVI · Multimodal | sliced Wasserstein distance: 0.08_{0.02} -> 0.05_{0.01} | Helped |
| Use PVI instead of the strongest baseline | SIVI · X-Shape | rejection rate: 0.11_{0.04} -> 0.06_{0.03} | Helped |
| Use PVI instead of the strongest baseline | SIVI · X-Shape | sliced Wasserstein distance: 0.10_{0.04} -> 0.07_{0.01} | Helped |
| Use PVI instead of the strongest baseline | Bayesian neural network · Concrete test | RMSE: 0.50_{0.03} -> 0.43_{0.03} | Helped |
| Use PVI instead of the strongest baseline | Bayesian neural network · Protein test | RMSE: 0.92_{0.04} -> 0.87_{0.05} | Helped |
| Use PVI instead of the strongest baseline | Bayesian neural network · Yacht test | RMSE: 0.17_{0.02} -> 0.13_{0.02} | Helped |
| Optimize the mixing distribution instead of fixing it near N(0,I_2) (PVIZero) (ablation) | PVI with the Skip kernel · Bimodal Gaussian mixture, μ=4 | multimodality captured | Helped |
| Replace the Push kernel with the Skip kernel (ablation) | PVI and PVIZero · Bimodal Gaussian mixture, μ=2 | bimodality captured | Helped |
| Replace the Push kernel with the Skip kernel (ablation) | PVI and PVIZero · Bimodal Gaussian mixture, increasing μ | mode collapse | Helped |
| Use the LSkip kernel instead of Skip (ablation) | PVI · Bimodal Gaussian mixture | multimodality captured | No clear effect |
Surprising findings
- The score matching SIVI baseline performs worst on the Bayesian neural network benchmarks (e.g., RMSE 0.98 on Yacht).
- Omitting the extra divergence term in the preconditioned mixing-distribution SDE has little effect in practice while reducing computational cost.
Limitations
- Theoretical results only hold for gamma > 0, whereas experiments use gamma = 0 corresponding to the regularized ELBO.
- Performance depends on kernel design; some kernels (e.g., skip) are more amenable to expressive mixing distributions.
Experimental setup
Datasets: waveform Concrete Protein Yacht
Benchmarks: toy density estimation (Banana, Multimodal, X-Shape) Bayesian logistic regression on waveform vs MCMC Bayesian neural network regression (Concrete, Protein, Yacht)
Metrics: rejection rate of two-sample kernel test sliced Wasserstein distance RMSE correlation coefficients vs MCMC
Baselines: UVI SVI SM (score matching SIVI) PVIZero
Builds on: semi-implicit variational inference Wasserstein gradient flow McKean-Vlasov SDE RMSProp
Closest papers by meaning
- Kernel Semi-Implicit Variational Inference (ICML 2024 · similarity 0.92)
- GAD-PVI: A General Accelerated Dynamic-Weight Particle-Based Variational Inference Framework (AAAI 2024 · similarity 0.91)
- Functional Gradient Flows for Constrained Sampling (NeurIPS 2024 · similarity 0.90)
- Revisiting Unbiased Implicit Variational Inference (ICML 2025 · similarity 0.90)
- Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians (ICML 2024 · similarity 0.89)
- Asymptotically exact variational flows via involutive MCMC kernels (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.