Research map / Probabilistic and scientific modeling / Variational and generative sampling
Generalized Variational Inference via Optimal Transport
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VOT, a black-box variational inference method that replaces the KL term with an entropy-regularized optimal transport distance between the approximate posterior and prior, with a gradually increased weight λ for over-parameterized models.
Problem
Standard variational inference uses KL-type divergences that are zero-forcing and metric-agnostic, leading to underestimated posterior variance and poor approximations, especially when the approximate posterior does not match the prior well.
Approach
Uses an entropy-regularized optimal transport (Sinkhorn-style semi-dual) distance as the regularizer between the variational distribution and prior, optimized via alternating dual computation and reparameterization-based gradients, plus a sigmoid schedule on the regularization weight λ.
Key results
- VOT achieves the lowest test RMSE on most UCI datasets, e.g., 4.7659 on concrete vs 5.3740 for alpha=0.5 and 5.5670 for chi-squared VI.
- On the over-parameterized setting, VOT with the lambda schedule outperforms lambda=1 on concrete (RMSE 4.9419 vs 5.1378) and estate (RMSE 8.5780 vs 10.6277).
- VOT produces qualitatively realistic MNIST reconstructions and smooth digit interpolations compared with WVI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OT distance in place of α-divergence regularization | VOT Bayesian neural network · UCI regression benchmarks | RMSE | Helped |
| Use OT distance in place of α-divergence regularization | VOT Bayesian neural network · UCI regression benchmarks | NLL | Mixed |
| Use the scheduled λ instead of fixing λ=1 (ablation) | VOT, two-hidden-layer Bayesian neural network · concrete | RMSE: 5.1378 -> 4.9419 | Helped |
| Use the scheduled λ instead of fixing λ=1 (ablation) | VOT, two-hidden-layer Bayesian neural network · estate | RMSE: 10.6277 -> 8.5780 | Helped |
| Use the scheduled λ instead of fixing λ=1 (ablation) | VOT, two-hidden-layer Bayesian neural network · aquatic | RMSE: 1.3901 -> 1.2768 | Helped |
| Use the scheduled λ instead of fixing λ=1 (ablation) | VOT, two-hidden-layer Bayesian neural network · concrete | NLL: 3.6230 -> 3.6076 | Helped |
| Use the scheduled λ instead of fixing λ=1 (ablation) | VOT, two-hidden-layer Bayesian neural network · estate | NLL: 3.9733 -> 3.6791 | Helped |
| Use the scheduled λ instead of fixing λ=1 (ablation) | VOT, two-hidden-layer Bayesian neural network · aquatic | NLL: 1.7101 -> 1.6782 | Helped |
Surprising findings
- On the concrete dataset, VOT has the best test RMSE but the worst test NLL (3.4900 vs about 3.11-3.17 for baselines).
Limitations
- OT computation is complex and VOT converges relatively slowly on some datasets.
- Convergence to a local optimum only; the optimization over the variational distribution is non-convex.
- Choice of ground cost and the lambda schedule constants are hand-set; the NLL is worse than baselines on some datasets.
Experimental setup
Datasets: UCI Concrete UCI Real Estate Valuation UCI Yacht Hydrodynamics QSAR Fish Toxicity QSAR Aquatic Toxicity UCI Airfoil Self-Noise UCI Combined Cycle Power Plant MNIST
Benchmarks: UCI regression datasets (RMSE, NLL) MNIST reconstruction and interpolation (qualitative)
Metrics: test RMSE test negative log likelihood marginal posterior variance
Baselines: alpha-divergence VI (alpha=0.5) KL-divergence VI alpha-divergence VI (alpha=1.5) chi-squared VI (alpha=2) WVI
Builds on: generalized variational inference alpha-divergence variational inference Wasserstein variational inference Sinkhorn entropy regularization
Closest papers by meaning
- Wasserstein Variational Inference (NeurIPS 2018 · similarity 0.92)
- Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.90)
- Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians (ICML 2024 · similarity 0.90)
- Model-Informed Flows for Bayesian Inference (NeurIPS 2025 · similarity 0.90)
- VIKING: Deep variational inference with stochastic projections (NeurIPS 2025 · similarity 0.89)
- Bayesian Online Natural Gradient (BONG) (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.