Research map / Image generation / Autoregressive and latent generation
Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EncDiff, which encodes an image into concept tokens that condition a latent diffusion U-Net via cross-attention, achieving unsupervised disentanglement without extra regularization.
Problem
Unsupervised disentangled representation learning usually needs tailored regularization losses or structural designs; it is unclear whether diffusion models can serve as a source of inductive bias for disentanglement.
Approach
Encode an image into per-factor concept tokens (one per latent dimension via non-shared MLPs) that condition a latent diffusion U-Net through cross-attention, trained end-to-end with the noise-prediction loss; the authors argue the diffusion process's time-varying information bottleneck and cross-attention act as inductive biases promoting disentanglement, with no extra regularizer.
Key results
- On Shapes3D, EncDiff reaches FactorVAE score 0.999 and DCI 0.969, versus DisDiff's 0.902 and 0.723.
- On MPI3D, EncDiff reports FactorVAE 0.872 and DCI 0.685, above DisDiff's 0.617 and 0.337.
- On Cars3D, EncDiff's DCI (0.279) is best, but its FactorVAE score (0.773) trails DisDiff (0.976); the authors note the Cars3D metrics are less reliable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EncDiff instead of DisDiff | disentangled representation learning · Shapes3D | DCI: 0.723 ± 0.013 -> 0.969 ± 0.030 | Helped |
| use EncDiff instead of DisDiff | disentangled representation learning · MPI3D | DCI: 0.337 ± 0.057 -> 0.685 ± 0.044 | Helped |
| use EncDiff instead of DisDiff | disentangled representation learning · Cars3D | FactorVAE score: 0.976 ± 0.018 -> 0.773 ± 0.060 | Hurt |
| add diffusion modelling as the decoder (ablation) | EncDiff on Shapes3D · Shapes3D | DCI: 0.178 ± 0.050 -> 0.969 ± 0.030 | Helped |
| use cross-attention instead of AdaGN (ablation) | EncDiff on Shapes3D · Shapes3D | DCI: 0.637 ± 0.068 -> 0.969 ± 0.030 | Helped |
| use cosine rather than linear variance schedule (ablation) | EncDiff on Shapes3D · Shapes3D | DCI: 0.930 ± 0.045 -> 0.969 ± 0.030 | Helped |
| use scalar-valued factors rather than vector-valued chunks (ablation) | EncDiff on Shapes3D · Shapes3D | DCI: 0.900 ± 0.045 -> 0.969 ± 0.030 | Helped |
| use 20 instead of 5 concept tokens (ablation) | EncDiff on Shapes3D · Shapes3D | DCI: 0.536 ± 0.073 -> 0.969 ± 0.030 | Helped |
| add orthogonality regularization (ablation) | EncDiff on Shapes3D · Shapes3D | DCI: 0.969 ± 0.030 -> 0.965 ± 0.040 | No clear effect |
| add sparsity regularization (ablation) | EncDiff on Shapes3D · Shapes3D | DCI: 0.969 ± 0.030 -> 0.972 ± 0.029 | No clear effect |
| replace the CNN encoder with a transformer encoder (ablation) | EncDiff on Shapes3D · Shapes3D | DCI: 0.969 ± 0.030 -> 0.898 ± 0.033 | Hurt |
| use the mapped vector-valued representation instead of scalar-valued representation (ablation) | EncDiff on Shapes3D · Shapes3D | DCI: 0.969 ± 0.030 -> 0.955 ± 0.030 | No clear effect |
Surprising findings
- Replacing cross-attention with AdaGN lowers DCI from 0.969 to 0.637, showing the conditioning mechanism matters a lot.
- Adding explicit orthogonality or sparsity regularization changes results only slightly (DCI 0.965 to 0.975), suggesting the architecture carries most of the disentanglement.
Limitations
- Performance on complex real-world data remains a challenge, as with other disentanglement methods.
- Sampling is still slower than VAE- and GAN-based methods.
- Cars3D quantitative metrics are unreliable because some factors are not labeled.
Experimental setup
Datasets: Shapes3D MPI3D Cars3D CelebA
Benchmarks: Shapes3D MPI3D Cars3D CelebA
Metrics: FactorVAE score DCI SSIM LPIPS MSE FID TAD
Baselines: FactorVAE beta-TCVAE InfoGAN-CR LD GANspace DisCo DisDiff InfoDiffusion Diff-AE PDAE
Builds on: latent diffusion models DisDiff Diff-AE AnnealVAE
Closest papers by meaning
- Can Diffusion Models Disentangle? A Theoretical Perspective (NeurIPS 2025 · similarity 0.90)
- Factorized Diffusion Autoencoder for Unsupervised Disentangled Representation Learning (AAAI 2024 · similarity 0.89)
- Why DiT Models Underperform as Representation Learners without Long Skip Connections (NeurIPS 2026 · similarity 0.89)
- Isometric Representation Learning for Disentangled Latent Space of Diffusion Models (ICML 2024 · similarity 0.88)
- Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning (ICML 2024 · similarity 0.88)
- Disentangled Representation Learning via Flow Matching (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.