Research map / Probabilistic and scientific modeling / Variational and generative sampling

Asymptotics of Alpha-Divergence Variational Inference Algorithms with Exponential Families

NeurIPS 2024 · Probabilistic and scientific modeling › Variational and generative sampling · approach: Annealing and divergence-based variational objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies the convergence of monotonic alpha-divergence variational inference for exponential families, proving geometric convergence for the exact algorithm and almost-sure convergence plus a law of the iterated logarithm for an unbiased sample-based variant.

Problem

Asymptotic convergence properties of alpha-divergence variational inference algorithms, especially when the iterations go to infinity and the algorithm is implemented with Monte Carlo estimates (which yield biased gradients).

Approach

Analyzes the monotonic alpha-divergence minimization scheme of prior work as a fixed-point/Minorize-Maximize iteration for exponential families, giving geometric rate conditions; proposes an unbiased empirical recursion (multiplying the biased estimator by the normalizer estimate) that is a Robbins-Monro procedure with almost-sure convergence and a law of the iterated logarithm, and links it to VA

Key results

• The exact monotonic algorithm converges to a strict local minimizer of the alpha-divergence at a geometric rate under assumptions H1-H4 (Theorem 1).
• The unbiased recursion converges almost surely to a critical point and satisfies a law of the iterated logarithm with rate sqrt(t ln S_t) (Theorems 2-3).
• On VAEs, the alpha-divergence-based VR and UB training outperform IWAE and VAE baselines on FID for suitable alpha (e.g., CIFAR-10 VR at alpha=0.99 gives 81.2 vs 122.2 for IWAE).

Measured effects reported in the paper

Surprising findings

• The unbiased (UB) gradient approach yields much higher losses than the normalized VR approach, and importance-weight normalization gives much more stable training than unbiased estimators.
• The best alpha differs between datasets and between the VR and UB approaches, with no consistent trend.

Limitations

• Theory covers only exponential variational families; general families are left to future work.
• The empirical (sample-based) monotonic algorithm is biased, so it lacks the convergence guarantees obtained for the unbiased variant.
• Convergence of the unbiased algorithm is conditional on a compact set containing iterates, and the moment and covariance assumptions may fail for heavy-tailed targets (e.g., Cauchy).

Experimental setup

Datasets: CIFAR-10 CelebA synthetic Gaussian mixture target Cauchy target

Benchmarks: CIFAR-10 CelebA

Metrics: alpha-divergence Fréchet Inception Distance (FID)

Baselines: MAX (biased maximization approach) SGE SGM NAT VAE IWAE

Builds on: monotonic alpha-divergence minimization (Daudel et al.) Variational Rényi bound Robbins-Monro stochastic approximation reparameterization trick

Closest papers by meaning

• Rényi Divergence Variational Inference (NeurIPS 2016 · similarity 0.90)
• Least squares variational inference (NeurIPS 2025 · similarity 0.88)
• Stochastic variance-reduced Gaussian variational inference on the Bures-Wasserstein manifold (ICLR 2025 · similarity 0.88)
• Generalized Variational Inference via Optimal Transport (AAAI 2024 · similarity 0.88)
• Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians (ICML 2024 · similarity 0.88)
• Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.87)