Research map / Probabilistic and scientific modeling / Variational and generative sampling
Forward $\chi^2$ Divergence Based Variational Importance Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes variational importance sampling (VIS), which maximizes an importance-sampled marginal log-likelihood estimate and fits the proposal by minimizing forward chi-square divergence in log space, evaluated on a toy mixture, VAEs on MNIST, and partially observable GLMs.
Problem
Maximizing the marginal log-likelihood of latent variable models with variational inference is biased: the ELBO can be high while the true marginal likelihood is low when posteriors are complex (multi-modal or heavy-tailed).
Approach
Use importance sampling with a proposal fit by minimizing forward chi-square divergence (estimated and optimized in log space via logsumexp and a pathwise or score-function gradient) so the log-IS marginal estimate is an asymptotically tighter lower bound than the ELBO and the proposal covers posterior modes.
Key results
- On the toy mixture model, VIS achieves better test marginal log-likelihood, CLL, and HLL than VI, CHIVI, and VBIS across 10 seeds, with more accurate parameter recovery.
- On MNIST VAE, VIS gives the highest test marginal log-likelihood over 20 epochs, above VI, CHIVI, VBIS, and IWAE.
- On the synthetic POGLM, VIS has the smallest weight error and the highest LL, CLL, and HLL, with VBIS and VIS having the smallest bias error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIS instead of VI, CHIVI, or VBIS | Toy mixture model · Toy mixture model test set | marginal log-likelihood | Helped |
| Use VIS instead of VI, CHIVI, or VBIS | Toy mixture model · Toy mixture model test set | complete log-likelihood | Helped |
| Use VIS instead of VI, CHIVI, or VBIS | Toy mixture model · Toy mixture model test set | hidden log-likelihood | Helped |
| Use VIS instead of IWAE, VI, CHIVI, or VBIS | VAE · MNIST test set | marginal log-likelihood | Helped |
| Use VIS instead of VI, CHIVI, or VBIS | POGLM · Synthetic POGLM test set | marginal log-likelihood | Helped |
| Use VIS instead of VI, CHIVI, or VBIS | POGLM · Synthetic POGLM test set | complete log-likelihood | Helped |
| Use VIS instead of VI, CHIVI, or VBIS | POGLM · Synthetic POGLM test set | hidden log-likelihood | Helped |
| Use VIS instead of VI, CHIVI, or VBIS | POGLM · Synthetic POGLM test set | weight estimation error | Helped |
| Use VIS instead of VI or CHIVI | POGLM · Synthetic POGLM test set | bias estimation error | Helped |
| Add hidden neurons and learn with VBIS or VIS rather than use a fully observed GLM | POGLM · RGC test segment | marginal log-likelihood | Helped |
| Use VIS instead of the three alternative methods | POGLM · RGC test segment | marginal log-likelihood | Helped |
| Use VIS instead of VI or VBIS | VAE · MNIST test set | marginal log-likelihood | Helped |
Surprising findings
- The score-function gradient estimator for minimizing forward chi-square is reported to outperform the pathwise estimator, contrary to the usual preference for pathwise gradients.
- With small Monte Carlo sample counts K, all sampling-based methods fail on the POGLM problem.
Limitations
- The authors note that the optimal forward chi-square proposal may not translate to practical benefit in real-world applications and needs further validation.
- Large numbers of Monte Carlo samples are required for good performance, increasing cost.
- The POGLM experiments show that hidden-to-hidden and hidden-to-visible weights are poorly recovered, reflecting the limitation of the proposal family.
Experimental setup
Datasets: MNIST synthetic toy mixture data synthetic POGLM spike trains retinal ganglion cell spike recordings (27 neurons)
Benchmarks: MNIST synthetic toy mixture synthetic POGLM RGC spike dataset
Metrics: marginal log-likelihood (LL) complete log-likelihood (CLL) hidden log-likelihood (HLL) parameter estimation error running time
Baselines: VI CHIVI VBIS IWAE
Builds on: variational inference importance weighted autoencoder CHIVI VBIS
Closest papers by meaning
- Variational Inference for Monte Carlo Objectives (ICML 2016 · similarity 0.92)
- Importance Weighted Hierarchical Variational Inference (NeurIPS 2019 · similarity 0.89)
- Differentiable Annealed Importance Sampling Minimizes The Jensen-Shannon Divergence Between Initial and Target Distribution (ICML 2024 · similarity 0.88)
- Least squares variational inference (NeurIPS 2025 · similarity 0.88)
- Generalized Variational Inference via Optimal Transport (AAAI 2024 · similarity 0.88)
- Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.