Research map / Probabilistic and scientific modeling / Variational and generative sampling
VISA: Variational Inference with Sequential Sample-Average Approximations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VISA, a forward-KL importance-weighted VI method that reuses fixed sample-average approximations within an ESS-based trust region, cutting model evaluations for expensive, possibly non-differentiable models.
Problem
Importance-weighted forward-KL variational inference needs fresh model evaluations at every gradient step, which is costly for computationally intensive, non-differentiable simulation-based models.
Approach
Adapt sample-average approximations to forward-KL importance-weighted VI: fix a sample set drawn from a proposal (the variational distribution at an earlier iterate), optimize the deterministic surrogate, and refresh samples only when the effective sample size of the weights between current and proposal falls below a threshold (a trust region).
Key results
- VISA achieves comparable forward-KL approximation accuracy to IWFVI on Lotka-Volterra with step sizes 0.001 and 0.005 while needing significantly fewer model evaluations to converge.
- The abstract reports computational savings of a factor two or more for conservatively chosen learning rates.
- On the Gaussian targets VISA fails to converge reliably at learning rate 0.05, while IWFVI still converges.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reuse samples with sequential sample-average approximations instead of refreshing them at every gradient step | VISA vs. IWFVI · High-dimensional Gaussians | convergence rate | Helped |
| use a learning rate of 0.05 | VISA · High-dimensional Gaussians | reliable convergence | Hurt |
| use lower ESS thresholds | VISA · High-dimensional Gaussians | convergence rate | Helped |
| use lower ESS thresholds, keeping samples fixed longer | VISA · High-dimensional Gaussians | posterior variance approximation | Hurt |
| reuse samples with sequential sample-average approximations instead of refreshing them at every gradient step | VISA vs. IWFVI · Lotka-Volterra | model evaluations to convergence | Helped |
| use a learning rate of 0.01 | VISA · Lotka-Volterra | model evaluations to convergence | Mixed |
| use a higher ESS threshold | VISA · Lotka-Volterra | test loss | Helped |
| reuse samples with sequential sample-average approximations | VISA vs. IWFVI · Pickover attractor | convergence stability | Helped |
| optimize the forward-KL SAA with L-BFGS and line search (ablation) | VISA · Forward-KL variational inference | optimization stability | Hurt |
Surprising findings
- VISA is more prone to underestimating posterior variance and overfitting to high-weight samples, especially at low ESS thresholds.
- L-BFGS, despite working for reparameterized SAA methods, amplified overfitting and caused collapsed variational distributions in the forward-KL setting.
Limitations
- Susceptible to bias and variance underestimation, especially with low ESS thresholds.
- Not well-suited to models with many latent variables or parameters, since few samples are used.
- L-BFGS and other second-order methods amplify overfitting in the forward-KL setting.
Experimental setup
Datasets: synthetic high-dimensional Gaussians simulated Lotka-Volterra observations simulated Pickover attractor observations
Benchmarks: Gaussian targets (D=128 diagonal, D=32 dense) Lotka-Volterra inference Pickover attractor inference
Metrics: symmetric KL divergence forward KL divergence evidence upper bound / test loss number of model evaluations
Baselines: IWFVI BBVI-SF BBVI-RP reweighted wake-sleep
Builds on: importance-weighted forward-KL VI reweighted wake-sleep reparameterized black-box VI with SAA batched L-BFGS pathfinder
Closest papers by meaning
- Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.87)
- Generalized Variational Inference via Optimal Transport (AAAI 2024 · similarity 0.87)
- Least squares variational inference (NeurIPS 2025 · similarity 0.86)
- Asymptotics of Alpha-Divergence Variational Inference Algorithms with Exponential Families (NeurIPS 2024 · similarity 0.86)
- Model-Informed Flows for Bayesian Inference (NeurIPS 2025 · similarity 0.86)
- Revisiting Unbiased Implicit Variational Inference (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.