Research map / Probabilistic and scientific modeling / Variational and generative sampling
EigenVI: score-based variational inference with orthogonal function expansions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EigenVI, a black-box variational inference method whose variational family is the squared weighted sum of orthogonal basis functions (e.g., normalized Hermite), fit by minimizing an importance-sampled Fisher divergence that reduces to a minimum eigenvalue problem.
Problem
Black-box variational inference is usually optimized with SGD on the ELBO, which is sensitive to learning rates and termination criteria, and Gaussian families are too restrictive for non-Gaussian posteriors.
Approach
Variational densities are squared weighted sums of orthonormal basis functions (lowest order gives a Gaussian); with a score-based Fisher divergence estimated by importance sampling, fitting the weights becomes a quadratic form whose minimizer is the minimal eigenvector of a K×K matrix, so no gradient descent is needed.
Key results
- On posteriordb models, EigenVI is reported to be more accurate than Gaussian BBVI methods (ADVI, GSM, BaM) in empirical Fisher divergence as the number of basis functions increases.
- EigenVI with the Hermite family fits multimodal, asymmetric and heavy-tailed 2D targets better than the Gaussian score-based VI, which is measured by forward KL.
- The eigenvalue solve is fast and the samples can be reused across different basis-function counts, with computation of the score matrix parallelized.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use higher-order Hermite basis functions instead of a Gaussian variational family | EigenVI · 2D synthetic targets | KL(p;q) | Helped |
| increase the number of basis functions | EigenVI · posteriordb hierarchical models | Fisher divergence | Helped |
| standardize the target before fitting | EigenVI with Hermite basis functions · R^D targets | basis functions required for a comparable fit: K=14 versus K=7 | Helped |
| use EigenVI instead of a full-covariance Gaussian BBVI method | EigenVI · posteriordb hierarchical models | posterior approximation accuracy | Helped |
| use EigenVI instead of a Gaussian variational family | EigenVI · 8-schools | posterior fit | Helped |
| use a real-NVP normalizing flow instead of EigenVI | posterior inference on 8-schools · 8-schools | posterior fit | No clear effect |
| use a non-standardized target instead of a standardized target | EigenVI with Hermite basis functions · Gaussian target with mean 3 and variance 1/8 | basis functions required for a comparable fit: 16th-order expansion | Hurt |
| increase the number of importance samples | EigenVI · synthetic targets | forward KL | Helped |
Limitations
- Number of basis functions grows exponentially with dimension for Cartesian-product families, limiting high-dimensional use.
- Relies on importance sampling with a hand-chosen proposal distribution and sample count B.
- Highly non-Gaussian targets may require very high-order expansions in the Hermite family.
Experimental setup
Datasets: posteriordb
Benchmarks: posteriordb sinh-arcsinh synthetic targets 2D synthetic mixture, funnel, cross targets
Metrics: forward KL empirical Fisher divergence wall-clock time
Baselines: ADVI Gaussian score matching (GSM) batch and match VI (BaM) real-NVP normalizing flow
Builds on: batch and match VI automatic differentiation VI Gaussian score matching real-NVP
Closest papers by meaning
- Least squares variational inference (NeurIPS 2025 · similarity 0.91)
- Batch and match: black-box variational inference with a score-based divergence (ICML 2024 · similarity 0.90)
- Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.90)
- Natural Gradient VI: Guarantees for Non-Conjugate Models (NeurIPS 2025 · similarity 0.90)
- Fisher meets Feynman: score-based variational inference with a product of experts (NeurIPS 2025 · similarity 0.89)
- Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inference (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.