Research map / Probabilistic and scientific modeling / Variational and generative sampling
Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDVI, a black-box variational inference method that uses a diffusion model in latent space as an expressive approximate posterior, trained with a regularized ELBO plus a wake-sleep-inspired diffusion loss.
Problem
Approximate posteriors in deep latent variable models are often too simple, loosening the ELBO; existing expressive posteriors (normalizing flows, GANs) have architectural or training limitations.
Approach
Parameterize the variational posterior as a latent-space denoising diffusion process that iteratively refines latents from a Gaussian, and train it with an ELBO augmented by a forward-KL wake-sleep style diffusion loss that avoids backpropagating through sampling and does not require invertible architectures.
Key results
- On MNIST unsupervised with the pinwheel prior, DDVI reaches ELBO -3.88 ± 0.96 versus -12.13 for AEVB and -4.19 for AEVB-IAF.
- On 1000 Genomes genotype clustering, DDVI attains the best cluster purity (0.45 ± 0.03) and NMI (0.66 ± 0.04) among compared methods.
- DDVI attains the best ELBO in semi-supervised MNIST across all three priors (e.g., -0.24 for pinwheel).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DDVI instead of AEVB-IAF | DGLVM · MNIST, pinwheel prior | ELBO: -4.19 \pm 0.05 -> -3.88 \pm 0.96 | Helped |
| use DDVI instead of AEVB-IAF | DGLVM · MNIST, swiss roll prior | ELBO: -5.10 \pm 0.30 -> -5.03 \pm 0.58 | Helped |
| use DDVI instead of AEVB-IAF | DGLVM · MNIST, square prior | ELBO: -3.97 \pm 0.22 -> -3.79 \pm 0.14 | Helped |
| use DDVI instead of AAEB | DGLVM · MNIST, pinwheel prior | MMD: 0.68 \pm 0.02 -> 0.67 \pm 0.04 | Helped |
| use DDVI instead of AEVB-IAF | DGLVM · MNIST, swiss roll prior | MMD: 0.61 \pm 0.15 -> 0.62 \pm 0.33 | Mixed |
| use DDVI instead of AEVB-IAF | DGLVM · MNIST, square prior | MMD: 0.75 \pm 0.12 -> 0.66 \pm 0.07 | Helped |
| use DDVI instead of PIS | DGLVM · MNIST, swiss roll prior | Latent NLL: 2.40 \pm 1.01 -> 3.86 \pm 0.17 | Hurt |
| use DDVI instead of AEVB-IAF | DGLVM · MNIST, pinwheel prior, 1,000 labels | ELBO: -2.10 \pm 0.26 -> -0.24 \pm 0.13 | Helped |
| use DDVI instead of AEVB-IAF | DGLVM · MNIST, swiss roll prior, 1,000 labels | ELBO: -5.38 \pm 1.78 -> -2.89 \pm 0.33 | Helped |
| use DDVI instead of AEVB-IAF | DGLVM · MNIST, square prior, 1,000 labels | ELBO: -2.67 \pm 0.83 -> 0.02 \pm 0.09 | Helped |
| use DDVI instead of AAEB | DGLVM · 1000 Genomes | Cluster Purity: 0.37 \pm 0.06 -> 0.45 \pm 0.03 | Helped |
| use DDVI instead of AAEB | DGLVM · 1000 Genomes | NMI: 0.63 \pm 0.02 -> 0.66 \pm 0.04 | Helped |
Surprising findings
- On the 1000 Genomes task, cluster completeness is not best for DDVI; AEVB has a higher mean completeness (0.78) though with overlapping confidence intervals.
Limitations
- Gains are reported mainly on small-scale synthetic priors and low-resolution data; likelihood competitiveness requires architecture improvements left out of scope.
- The simplified latent-space wake-sleep variant is not a tight bound on log p(x).
- The regularized objective differs from the ELBO and may not help on all models.
Experimental setup
Datasets: MNIST CIFAR-10 1000 Genomes
Benchmarks: MNIST CIFAR-10 1000 Genomes
Metrics: ELBO MMD latent NLL KNN accuracy cluster purity cluster completeness NMI
Baselines: AEVB AEVB-IAF AAEB PIS H-AEVB PCA t-SNE UMAP
Builds on: auto-encoding variational Bayes wake-sleep denoising diffusion probabilistic models inverse autoregressive flow path integral sampler
Closest papers by meaning
- Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference (ICML 2024 · similarity 0.90)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.89)
- EVODiff: Entropy-aware Variance Optimized Diffusion Inference (NeurIPS 2025 · similarity 0.89)
- DiffEnc: Variational Diffusion with a Learned Encoder (ICLR 2024 · similarity 0.89)
- Interpreting and Improving Diffusion Models from an Optimization Perspective (ICML 2024 · similarity 0.89)
- O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.