Research map / Probabilistic and scientific modeling / Variational and generative sampling
Kernel Semi-Implicit Variational Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KSIVI, which solves SIVI-SM's inner maximization in closed form using an RKHS so the objective becomes a computable kernel Stein discrepancy for semi-implicit variational inference.
Problem
Semi-implicit variational inference (SIVI) trained with ELBO surrogates is biased, and the score-matching variant SIVI-SM needs an additional inner-loop adversarial optimization that is unstable and expensive.
Approach
Optimize the Fisher-divergence minimax objective over an RKHS so the lower-level problem has an explicit solution; the upper-level objective becomes the squared kernel Stein discrepancy, which is estimated with samples from the hierarchical variational family using the denoising-score-matching trick (conditional scores only).
Key results
- On the conditioned diffusion process (dim 50/100/200), KSIVI (vanilla) reaches sliced-WD 0.0140/0.0115/0.0493 versus SIVI-SM 0.0917/0.0640/0.0628 and SIVI 0.1266/0.0981/0.0756.
- Per 10k iterations at dimension 100, KSIVI takes 90.48 s versus SIVI-SM 128.13 s and SIVI 88.12 s.
- On UCI BNN benchmarks KSIVI is on par with SIVI and SIVI-SM (e.g., Boston RMSE 2.555 vs 2.785 for SIVI-SM), without a clear overall advantage.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KSIVI instead of SIVI | Bayesian neural network · UCI benchmarks: Boston, Concrete, Power, Wine, Yacht, Protein | Test RMSE | Mixed |
| Use KSIVI instead of SIVI-SM | Bayesian neural network · UCI benchmarks: Boston, Concrete, Power, Wine, Yacht, Protein | Test NLL | Mixed |
| Use KSIVI instead of SIVI-SM | KSIVI · Conditioned diffusion, dimensions 50 and 100 | Sliced-WD: 0.0917 -> 0.0140; 0.0640 -> 0.0115 | Helped |
| Use KSIVI instead of SIVI-SM | KSIVI · Conditioned diffusion, dimension 200 | Sliced-WD: 0.0628 -> 0.0493 | Helped |
| Replace Gaussian kernel with Reisz kernel (ablation) | KSIVI · Product of two Student’s t-distributions, edge width 5 | Wasserstein distance: 0.1048 ± 0.03 -> 0.0451 ± 0.01 | Helped |
| Replace Gaussian kernel with Reisz kernel (ablation) | KSIVI · Product of two Student’s t-distributions, edge width 8 | Wasserstein distance: 0.1976 ± 0.08 -> 0.0816 ± 0.04 | Helped |
| Replace Gaussian kernel with Reisz kernel (ablation) | KSIVI · Product of two Student’s t-distributions, edge width 10 | Wasserstein distance: 0.2546 ± 0.09 -> 0.1136 ± 0.05 | Helped |
| Use KSIVI instead of SIVI-SM | KSIVI · Conditioned diffusion | Training time per 10,000 iterations: 70.42 vs 56.67; 128.13 vs 90.48; 149.61 vs 107.84 | Helped |
| Use the U-statistic gradient estimator instead of the vanilla estimator (ablation) | KSIVI · Conditioned diffusion, dimensions 50, 100, and 200 | Training time per 10,000 iterations: 56.67 -> 39.09; 90.48 -> 58.13; 107.84 -> 64.17 | Helped |
Surprising findings
- KSIVI does not show a significant advantage over SIVI and SIVI-SM on Bayesian neural network predictive metrics, which the authors attribute partly to the metrics.
- On the heavy-tailed Student-t ablation, MIED outperforms KSIVI at small edge widths, and a Riesz kernel helps KSIVI more than the Gaussian kernel.
Limitations
- KSD-based stationary points may differ from the target distribution, especially in high-dimensional non-convex settings.
- The Gaussian kernel may be less effective in high dimensions due to fast-decaying tails.
- Convergence theory relies on strong assumptions such as Lipschitz smoothness of the neural network mapping.
Experimental setup
Datasets: Banana Multimodal X-shaped waveform UCI regression datasets (Boston, Concrete, Power, Wine, Yacht, Protein) conditioned diffusion process (synthetic)
Benchmarks: Banana Multimodal X-shaped waveform Bayesian logistic regression conditioned diffusion process UCI Bayesian neural network regression
Metrics: KL divergence MMD sliced Wasserstein distance Wasserstein distance test RMSE test NLL training time pairwise correlation error
Baselines: SIVI SIVI-SM SGLD UIVI MIED KSDD
Builds on: SIVI SIVI-SM kernel Stein discrepancy denoising score matching UIVI
Closest papers by meaning
- Revisiting Unbiased Implicit Variational Inference (ICML 2025 · similarity 0.93)
- Particle Semi-Implicit Variational Inference (NeurIPS 2024 · similarity 0.92)
- Least squares variational inference (NeurIPS 2025 · similarity 0.88)
- EigenVI: score-based variational inference with orthogonal function expansions (NeurIPS 2024 · similarity 0.88)
- Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.88)
- Globally Convergent Variational Inference (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.