Research map / Probabilistic and scientific modeling / Variational and generative sampling
Learning Latent Space Hierarchical EBM Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a sequence of conditional EBM priors over a uni-scale y-space of a hierarchical VAE via a diffusion scheme, easing MCMC sampling and improving generation quality over joint and NCP-VAE EBM priors.
Problem
Multi-layer (hierarchical) generator models with Gaussian priors suffer from the prior hole problem; a single marginal EBM prior over multi-layer latent variables is highly multi-modal and hard to sample with MCMC, so EBM learning is ineffective.
Approach
Replace the marginal latent EBM prior with a sequence of conditional EBM priors trained along a diffusion process defined on a uni-scale y-space obtained by inverting the hierarchical Gaussian prior's reparameterization; each conditional EBM only matches slightly perturbed samples, so Langevin sampling starts from the noisy next-step sample and is easier.
Key results
- Achieves FID 8.93 and IS 9.03 on CIFAR-10 with T=3, versus 11.34 FID for Joint-EBM and 9.58 for DRL-EBM.
- Reaches FID 8.78 on CelebA-HQ-256 and 7.34 on LSUN-Church-64, improving over Joint-EBM (9.89, 8.38) and NCP-VAE (24.79).
- With a 3-layer HVAE backbone, FID improves from 81.44 to 35.13.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the Gaussian prior with the proposed diffusion EBM prior | NVAE on CIFAR-10 · CIFAR-10 | FID: 37.73 -> 8.93 | Helped |
| use the proposed diffusion EBM prior instead of Joint-EBM | NVAE on CIFAR-10 · CIFAR-10 | FID: 11.34 -> 8.93 | Helped |
| use the proposed diffusion EBM prior instead of Joint-EBM | NVAE on CelebA-HQ-256 · CelebA-HQ-256 | FID: 9.89 -> 8.78 | Helped |
| use the proposed diffusion EBM prior instead of Joint-EBM | NVAE on LSUN-Church-64 · LSUN-Church-64 | FID: 8.38 -> 7.34 | Helped |
| increase diffusion steps from T=3 to T=6 (ablation) | proposed EBM prior on CIFAR-10 · CIFAR-10 | FID: 8.93 -> 8.13 | Helped |
| increase diffusion steps from T=3 to T=6 (ablation) | proposed EBM prior on CIFAR-10 · CIFAR-10 | Time (seconds): 94.23 -> 193.45 | Hurt |
| increase Langevin steps from K=30 to K=50 (ablation) | proposed EBM prior on CIFAR-10 · CIFAR-10 | FID: 9.98 -> 8.93 | Helped |
| increase Langevin steps from K=50 to K=100 (ablation) | proposed EBM prior on CIFAR-10 · CIFAR-10 | FID: 8.93 -> 8.13 | Helped |
| increase Langevin steps from K=50 to K=100 (ablation) | proposed EBM prior on CIFAR-10 · CIFAR-10 | Time (seconds): 94.23 -> 166.34 | Hurt |
| add the proposed EBM prior to the 3-layer HVAE backbone (ablation) | HVAE on CIFAR-10 · CIFAR-10 | FID: 81.44 -> 35.13 | Helped |
Limitations
- Performance remains inferior to state-of-the-art diffusion models.
- Sampling requires iterative Langevin dynamics, which is slow.
Experimental setup
Datasets: CIFAR-10 CelebA-HQ-256 LSUN-Church-64 CelebA-64 SVHN
Benchmarks: CIFAR-10 CelebA-HQ-256 LSUN-Church-64 SVHN OOD detection
Metrics: FID IS AUROC sampling time
Baselines: Joint-EBM NCP-VAE DRL-EBM NVAE HVAE BIVA VAEBM Adv-EBM NCSN LSGM DDPM StyleGANv2
Builds on: NVAE Joint EBM prior NCP-VAE Diffusion Recovery Likelihood EBM
Closest papers by meaning
- Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood (ICLR 2024 · similarity 0.88)
- Probing the Latent Hierarchical Structure of Data via Diffusion Models (ICLR 2025 · similarity 0.87)
- Improving Adversarial Energy-Based Model via Diffusion Process (ICML 2024 · similarity 0.87)
- Posterior Inference in Latent Space for Scalable Constrained Black-box Optimization (NeurIPS 2026 · similarity 0.87)
- Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modelling (NeurIPS 2024 · similarity 0.86)
- Outsourced Diffusion Sampling: Efficient Posterior Inference in Latent Spaces of Generative Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.