Research map / Probabilistic and scientific modeling / Variational and generative sampling

Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders

ICLR 2024 · Probabilistic and scientific modeling › Variational and generative sampling · approach: Regularizing VAE latent information · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretically characterizes posterior collapse in linear conditional VAE and two-level linear hierarchical VAE via global minima, and validates the insights empirically on linear and nonlinear (MLP/CNN/ResNet) models on MNIST and CIFAR10.

Problem

Posterior collapse in VAEs is poorly understood theoretically beyond the standard VAE; conditional VAE and hierarchical VAE have not been analyzed.

Approach

Derives closed-form global minima of linear CVAE and two-latent linear Markovian HVAE objectives, showing collapse is driven by input-output correlation (CVAE) and learnable encoder variance together with beta hyperparameters (HVAE), and checks that these predictions transfer to nonlinear networks.

Key results

• In linear CVAE, the theory predicts complete collapse at beta=3.33 for the top singular value, matching training behavior; higher x-y correlation yields higher KL and AU (e.g., KL 80.37 / AU 67.2% for independent x vs. 0.01 / 0.1% for x=y).
• On MNIST, 3 of 5 latent dimensions collapse with learnable encoder variance, whereas unlearnable variance avoids collapse; ReLU VAE AU rises from 69% (learnable) to 100% (unlearnable).
• Decreasing beta2 and eta_dec reduces collapse in ReLU MHVAE, while decreasing beta1 increases collapse, consistent with the theorem.

Measured effects reported in the paper

Surprising findings

• Learnable encoder variance is a cause of posterior collapse, contrary to a prior claim that it is not.
• Increasing beta1 mitigates collapse of the second latent in the hierarchical VAE, which is counterintuitive.

Limitations

• Theory covers only linear CVAE and two-latent MHVAE; the case with both encoder variances learnable in MHVAE is left open.
• Analysis characterizes global minima and does not address training dynamics.

Experimental setup

Datasets: MNIST CIFAR10 synthetic Gaussian data

Benchmarks: MNIST CIFAR10

Metrics: (epsilon, delta)-collapse KL divergence active units (AU) importance-weighted log-likelihood mean absolute difference of singular values

Baselines: linear VAE with learnable encoder variance ReLU CVAE ReLU MHVAE

Builds on: VAE beta-VAE linear VAE posterior collapse analysis (Lucas et al.) probabilistic PCA

Closest papers by meaning

• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (ICLR 2019 · similarity 0.94)
• Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (NeurIPS 2019 · similarity 0.93)
• Posterior Collapse and Latent Variable Non-identifiability (NeurIPS 2021 · similarity 0.93)
• The Usual Suspects? Reassessing Blame for VAE Posterior Collapse (ICML 2020 · similarity 0.93)
• Preventing Posterior Collapse with delta-VAEs (ICLR 2019 · similarity 0.92)
• Be a Goldfish: Forgetting Bad Conditioning in Sparse Linear Regression via Variational Autoencoders (ICML 2025 · similarity 0.90)