Research map / Probabilistic and scientific modeling / Variational and generative sampling
SoftCVI: Contrastive variational inference with self-generated soft labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SoftCVI, which derives variational objectives from contrastive estimation by classifying proposal samples against soft labels computed from the unnormalized posterior, and reports better calibration and forward-KL than ELBO and SNIS-fKL on Bayesian inference tasks.
Problem
Variational inference for Bayesian posteriors known only up to a normalizing constant is often unreliable: reverse-KL (ELBO) is mode-seeking and mass-covering alternatives are biased or unstable to train.
Approach
Reframe variational fitting as a softmax classification task over K samples from the variational distribution, where ground-truth soft labels are computed from the unnormalized posterior and a tempered negative distribution; the objective has zero-variance gradient at the optimum and with pi=p^- =q reduces to SNIS-fKL plus a control variate.
Key results
- SoftCVI with alpha=0.75 generally gave the best calibrated posteriors and highest mass on reference samples across tasks compared to ELBO and SNIS-fKL.
- On the multimodal SLCP task only the SoftCVI objectives performed well, while ELBO often missed modes.
- SoftCVI with alpha=1 matches SNIS-fKL in expectation but yields better calibration and mass coverage on most tasks, with the gap larger for small K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SoftCVI with α=0.75 instead of ELBO | Bayesian neural additive model · Bayesian neural additive model, widths 50 and 100 | test log-likelihood: -2.392 ± 0.005 -> -2.424 ± 0.007; -2.392 ± 0.005 -> -2.423 ± 0.005 | Hurt |
| Use SoftCVI with α=1 instead of SNIS-fKL | Bayesian neural additive model · Bayesian neural additive model | test log-likelihood: -2.422 ± 0.007 vs -2.422 ± 0.007; -2.423 ± 0.005 vs -2.421 ± 0.005 | No clear effect |
| Use SoftCVI with α=0.75 instead of α=1 (ablation) | SoftCVI · Bayesian neural additive model | test log-likelihood: -2.424 ± 0.007 vs -2.422 ± 0.007; -2.423 ± 0.005 vs -2.423 ± 0.005 | No clear effect |
| Use SoftCVI with α=0.75 instead of α=1 (ablation) | SoftCVI · Bayesian neural additive model, width 100 | prediction IQR: 0.983 ± 0.017 vs 0.943 ± 0.012 | Mixed |
| Use the ELBO instead of SoftCVI or SNIS-fKL | Bayesian neural additive model · Bayesian neural additive model | test log-likelihood: -2.422 ± 0.007 -> -2.392 ± 0.005; -2.424 ± 0.007 -> -2.392 ± 0.005 | Helped |
| Use a flat negative distribution (α=0) instead of α=0.75 (ablation) | SoftCVI · SLCP | posterior mass leakage | Hurt |
| Use SoftCVI with α=0.75 instead of SNIS-fKL | Bayesian inference tasks · Across reported tasks | posterior calibration | Helped |
| Use SoftCVI with α=1 instead of SNIS-fKL | Bayesian inference tasks · Eight schools | posterior calibration | No clear effect |
| Use SoftCVI with α=1 instead of SNIS-fKL | Bayesian inference tasks · Tasks other than eight schools | mass on reference posterior samples | Helped |
| Use SoftCVI instead of ELBO or SNIS-fKL | Flexible variational inference · SLCP | mode coverage | Helped |
| Use ELBO instead of SoftCVI | SLCP · SLCP | mode coverage | Hurt |
| Use SNIS-fKL instead of SoftCVI | Flexible variational inference · SLCP | mode approximation | Hurt |
Surprising findings
- Using a flat negative distribution (alpha=0) leads to mass leakage into regions of negligible posterior density.
- SNIS-fKL performance degrades substantially when K is small (K=2).
Limitations
- Requires an explicitly defined variational density (cannot use implicit models directly).
- Choice of negative distribution (alpha) is sensitive; low alpha causes high variance and degenerate labels in high dimensions.
- Gaussian variational family is misspecified for Bayesian neural network posteriors, limiting the variance-reduction benefit.
Experimental setup
Datasets: PosteriorDB reference posteriors SBI-Benchmark SLCP reference posteriors synthetic linear regression data synthetic GARCH and eight schools data
Benchmarks: eight schools linear regression SLCP GARCH(1,1) Bayesian neural additive model
Metrics: coverage probability log probability of reference posterior samples posterior mean accuracy gradient signal-to-noise ratio test log-likelihood
Baselines: ELBO SNIS-fKL
Builds on: self-normalized importance sampling forward KL InfoNCE sticking the landing estimator masked autoregressive flow
Closest papers by meaning
- A Contrastive Divergence for Combining Variational Inference and MCMC (ICML 2019 · similarity 0.91)
- Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.89)
- Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians (ICML 2024 · similarity 0.88)
- Generalized Variational Inference via Optimal Transport (AAAI 2024 · similarity 0.88)
- Maxitive Donsker-Varadhan Formulation for Possibilistic Variational Inference (NeurIPS 2026 · similarity 0.88)
- EigenVI: score-based variational inference with orthogonal function expansions (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.