Research map / Probabilistic and scientific modeling / Variational and generative sampling
Globally Convergent Variational Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that minimizing the amortized expected forward KL (neural posterior estimation) converges globally in the infinite-width NTK limit, unlike ELBO optimization which gets stuck in local optima.
Problem
Variational inference with the ELBO is nonconvex and only guarantees convergence to local optima; the amortized expected forward KL (neural posterior estimation) lacks global convergence guarantees.
Approach
Analyzes the amortized expected forward KL objective as a functional that is strictly convex over an RKHS when the variational family is an exponential family, then uses the neural tangent kernel and uniform kernel convergence to show gradient descent on the network parameters approaches the unique functional minimizer in the infinite-width limit.
Key results
- On the toy rotation example, linearized and full networks have nearly identical held-out NLL at width 1024, indicating the asymptotic NTK regime is approached at finite width.
- In amortized clustering, ELBO optimization converges to label-switched local optima, whereas minimizing the expected forward KL converges to the correctly ordered global optimum without label switching.
- On rotated MNIST, IWBO optimization often gets stuck near shallow local optima, while minimizing the expected forward KL converges to the correct 260-degree angle from all initializations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| minimize expected forward KL (L_P) instead of ELBO | amortized clustering inference for cluster centers Z · Label Switching in Amortized Clustering | average L1 distance to true cluster centers | Helped |
| minimize expected forward KL (L_P) instead of ELBO | amortized clustering inference for cluster centers Z · Label Switching in Amortized Clustering | proportion of trials with correctly ordered cluster centers | Helped |
| minimize expected forward KL (L_P) instead of maximize IWBO | rotated MNIST angle inference · Rotated MNIST Digits | angle point estimate | Helped |
| maximize IWBO rather than minimize expected forward KL (L_P) | rotated MNIST angle inference · Rotated MNIST Digits | angle point estimate | Mixed |
| minimize expected forward KL instead of negative ELBO | amortized Gaussian inference for rotation angle · Local Optima vs. Global Optima | forward KL | Helped |
| minimize expected forward KL instead of negative ELBO | amortized Gaussian inference for rotation angle · Local Optima vs. Global Optima | reverse KL | Helped |
| minimize expected forward KL instead of negative ELBO | amortized Gaussian inference for rotation angle · Local Optima vs. Global Optima | negative log-likelihood | Helped |
| minimize expected forward KL instead of negative ELBO | amortized Gaussian inference for rotation angle · Local Optima vs. Global Optima | angle point estimate | Helped |
| increase network width | the two-layer ReLU encoder and its linearization · Toy Example | held-out negative log-likelihood difference between network and linearization | Helped |
Surprising findings
- A likelihood-free objective (expected forward KL) can outperform likelihood-based ELBO optimization even when the likelihood is available.
Limitations
- Global convergence proven only for an infinite-width, two-layer ReLU network with a fixed positive-definite NTK and compact data space.
- Conditions such as positive-definite limiting NTK are assumed rather than verified in general; experiments on other architectures are empirical only.
- Expected forward KL requires sampling from the joint model, so it is only applicable to amortized inference settings in the stated form.
Experimental setup
Datasets: MNIST synthetic data from generative models
Metrics: held-out negative log-likelihood l1 distance to true cluster centers proportion of correctly ordered cluster centers forward KL reverse KL angle point estimate
Baselines: ELBO optimization importance-weighted bound (IWBO)
Builds on: neural tangent kernel neural posterior estimation reweighted wake-sleep lazy training
Closest papers by meaning
- Brain-like Variational Inference (NeurIPS 2025 · similarity 0.90)
- VIKING: Deep variational inference with stochastic projections (NeurIPS 2025 · similarity 0.89)
- Understanding Stochastic Natural Gradient Variational Inference (ICML 2024 · similarity 0.88)
- Bayesian Online Natural Gradient (BONG) (NeurIPS 2024 · similarity 0.88)
- Generalized Variational Inference via Optimal Transport (AAAI 2024 · similarity 0.88)
- Variational Inference for SDEs Driven by Fractional Noise (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.