Research map / Probabilistic and scientific modeling / Variational and generative sampling
Why do Variational Autoencoders Really Promote Disentanglement?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper gives a theoretical account of why VAEs disentangle: using a locally non-linear decoder model, it shows that minimizing the stochastic part of the reconstruction loss under the KL term promotes orthogonal decoder columns, which in turn promotes disentanglement, supported by experiments on four datasets.
Problem
Prior explanations of VAE-based disentanglement rely on local linearity or linearized decoders (and PCA analogies), which do not capture real non-linear VAEs; the link between decoder orthogonality and disentanglement had only been shown experimentally.
Approach
Models the VAE decoder locally as a composition of a linear map M_D and a non-linear function, shows that minimizing the stochastic reconstruction loss (with fixed deterministic loss and KL) forces columns of M_D to be orthogonal, and proves that this orthogonality yields lower KL and hence disentanglement.
Key results
- Non-linear local decoder approximation has lower error than the linear approximation of Zietlow et al. 2021 across all datasets and VAE variants (e.g., dSprites beta-TCVAE 0.8502 vs 0.9209).
- Across VAE, beta-VAE and beta-TCVAE on four datasets, lower OD-Score (more orthogonal decoder) corresponds to higher MIG and MIG-Sup scores.
- Theory proves that every local minimum of the constrained stochastic reconstruction loss is a global minimum, and that global minima have orthogonal decoder columns with variance inversely proportional to column norm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a nonlinear rather than linear local decoder approximation | β-TCVAE, β-VAE, and VAE · dSprites | approximation error: β-TCVAE: 0.9209 -> 0.8502; β-VAE: 0.8394 -> 0.7829; VAE: 0.4693 -> 0.4166 | Helped |
| Use a nonlinear rather than linear local decoder approximation | β-TCVAE, β-VAE, and VAE · 3D Faces | approximation error: β-TCVAE: 0.8679 -> 0.8088; β-VAE: 0.8080 -> 0.7694; VAE: 0.4956 -> 0.4236 | Helped |
| Use a nonlinear rather than linear local decoder approximation | β-TCVAE, β-VAE, and VAE · 3DShapes | approximation error: β-TCVAE: 0.8848 -> 0.8122; β-VAE: 0.8244 -> 0.7944; VAE: 0.4856 -> 0.4266 | Helped |
| Use a nonlinear rather than linear local decoder approximation | β-TCVAE, β-VAE, and VAE · MPI3D | approximation error: β-TCVAE: 0.9646 -> 0.8624; β-VAE: 0.5240 -> 0.8043; VAE: 0.5024 -> 0.4832 | Mixed |
| Model the local decoder as nonlinear instead of linearizing the stochastic reconstruction loss | β-TCVAE, β-VAE, and VAE · dSprites | approximation error: β-TCVAE: 0.4866 -> 0.4532**; β-VAE: 0.5243 -> 0.4907*; VAE: 0.6680 -> 0.6643 | Helped |
| Model the local decoder as nonlinear instead of linearizing the stochastic reconstruction loss | β-TCVAE, β-VAE, and VAE · 3D Faces | approximation error: β-TCVAE: 0.5690 -> 0.5498*; β-VAE: 0.5799 -> 0.5296*; VAE: 0.6039 -> 0.5887** | Helped |
| Model the local decoder as nonlinear instead of linearizing the stochastic reconstruction loss | β-TCVAE, β-VAE, and VAE · 3DShapes | approximation error | Helped |
| Model the local decoder as nonlinear instead of linearizing the stochastic reconstruction loss | β-TCVAE, β-VAE, and VAE · MPI3D | approximation error | Helped |
| Relate orthogonality of the local decoder matrix to disentanglement | VAE, β-VAE, and β-TCVAE · dSprites, 3D Faces, 3DShapes, and MPI3DComplex | MIG and MIG-Sup | Helped |
Surprising findings
- Columns of the linear part of the decoder being orthogonal does not imply orthogonal columns of the full non-linear local decoder f_D(M_D).
Limitations
- Analysis relies on a polarized-regime assumption and a local Taylor-style approximation of the decoder.
- The alignment of latent axes with the standard basis is argued but left as an open question.
- Experiments are limited to image datasets and three VAE variants with MIG-based evaluation.
Experimental setup
Datasets: dSprites 3D Faces 3D Shapes MPI3D-Complex
Benchmarks: dSprites 3D Faces 3D Shapes MPI3D-Complex
Metrics: MIG MIG-Sup OD-Score (orthogonality deviation score) approximation error
Baselines: Rolinek et al. 2019 linearized approximation Zietlow et al. 2021 local linear approximation
Builds on: beta-VAE beta-TCVAE Rolinek et al. 2019 VAE PCA analysis Zietlow et al. 2021 linear VAE PCA Kingma & Welling VAE
Closest papers by meaning
- Variational Autoencoders Pursue PCA Directions (by Accident) (CVPR 2019 · similarity 0.91)
- Rethinking Controllable Variational Autoencoders (CVPR 2022 · similarity 0.89)
- An Identifiable Double VAE For Disentangled Representations (ICML 2021 · similarity 0.88)
- $\alpha$TC-VAE: On the relationship between Disentanglement and Diversity (ICLR 2024 · similarity 0.87)
- Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders (ICLR 2024 · similarity 0.86)
- Purify Unlearnable Examples via Rate-Constrained Variational Autoencoders (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.