Research map / Probabilistic and scientific modeling / Variational and generative sampling
Revisiting Unbiased Implicit Variational Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits unbiased implicit VI by replacing its inner MCMC with importance sampling from a conditional normalizing flow trained via expected forward KL, yielding AISIVI (and BSIVI as a baseline) that match or beat KSIVI/PVI on SIVI benchmarks.
Problem
Semi-implicit variational inference needs accurate score gradients of the semi-implicit marginal; existing UIVI relies on expensive, biased-in-practice inner MCMC, so it has been dismissed as imprecise and computationally prohibitive.
Approach
Replace UIVI's MCMC conditional sampler with importance sampling from a conditional normalizing flow proposal, trained by minimizing the expected forward KL(q(eps|z) || tau(eps|z)); the optimal proposal equals the true conditional, which debiases the score-gradient estimator, and the reverse-KL ELBO is optimized via the path gradient.
Key results
- On the conditioned diffusion process, AISIVI reaches log marginal likelihood 74062, close to KSIVI's 74521, while IWHVI reaches 67667 and UIVI 40207.
- On toy Banana density, AISIVI achieves KL 0.0853 versus 0.3022 for BSIVI.
- On Bayesian logistic regression, AISIVI and BSIVI perform nearly equally to PVI and KSIVI in marginal and pairwise density estimates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AISIVI instead of BSIVI (importance sampling rather than the base score estimator) (ablation) | SIVI · Banana | KL(p, q): 0.3022 -> 0.0853 | Helped |
| Use AISIVI instead of BSIVI (importance sampling rather than the base score estimator) (ablation) | SIVI · Multimodal | KL(p, q): 0.0017 -> 0.0044 | Hurt |
| Use AISIVI instead of BSIVI (importance sampling rather than the base score estimator) (ablation) | SIVI · X-shape | KL(p, q): 0.0034 -> 0.0072 | Hurt |
| Use AISIVI rather than KSIVI | SIVI on the conditioned diffusion process · Conditioned diffusion process | Log ML: 74521 -> 74062 | Hurt |
| Use importance sampling with the learned proposal (AISIVI) rather than the base score estimator (BSIVI) (ablation) | SIVI · Conditioned diffusion process | Log ML: 60556 -> 74062 | Helped |
| Use AISIVI rather than IWHVI's joint-training approach (ablation) | SIVI on the conditioned diffusion process · Conditioned diffusion process | Log ML: 67667 -> 74062 | Helped |
| Use AISIVI rather than PVI | SIVI on the conditioned diffusion process · Conditioned diffusion process | Log ML: 53121 -> 74062 | Helped |
| Use AISIVI rather than UIVI's MCMC-based approach | SIVI on the conditioned diffusion process · Conditioned diffusion process | Log ML: 40207 -> 74062 | Helped |
Surprising findings
- The plain Monte Carlo score estimator with uninformed epsilon samples (BSIVI) performs surprisingly well despite being biased.
Limitations
- The method lacks inherent exploration, which may limit modeling of multimodal distributions; temperature annealing is suggested but not principled.
- Choice of normalizing flow architecture is a single affine coupling design; other flow combinations were not explored.
- UIVI comparison was limited to an inner batch size of 2 due to computational constraints.
Experimental setup
Datasets: WAVEFORM
Benchmarks: Banana X-shape Multimodal WAVEFORM Bayesian logistic regression conditioned diffusion process
Metrics: KL divergence pairwise correlation coefficients log marginal likelihood
Baselines: KSIVI PVI IWHVI UIVI SGLD
Builds on: unbiased implicit variational inference semi-implicit variational inference RealNVP importance weighted hierarchical variational inference
Closest papers by meaning
- Kernel Semi-Implicit Variational Inference (ICML 2024 · similarity 0.93)
- Particle Semi-Implicit Variational Inference (NeurIPS 2024 · similarity 0.90)
- Natural Gradient VI: Guarantees for Non-Conjugate Models (NeurIPS 2025 · similarity 0.88)
- Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.88)
- EigenVI: score-based variational inference with orthogonal function expansions (NeurIPS 2024 · similarity 0.88)
- SoftCVI: Contrastive variational inference with self-generated soft labels (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.