Research map / Image generation / Autoregressive and latent generation
Compositional Image Decomposition with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomp Diffusion decomposes a single image into K latent-conditioned diffusion denoisers (trained by reconstruction) whose composition yields recombinable global and local factors, outperforming COMET and object-centric baselines on reconstruction.
Problem
Unsupervised compositional decomposition of images into reusable factors (global like lighting/expression, or local like objects) that can be recombined within and across datasets and models; prior energy-based COMET is unstable and blurry.
Approach
Replace COMET's energy functions with diffusion denoisers: an encoder infers K latents from an image, each conditioning a denoising network whose summed noise predictions reconstruct the image; trained with a direct x0-prediction denoising loss, enabling stable single-step training and composition across models.
Key results
- On 64x64 reconstruction across four datasets, Decomp Diffusion achieves the best FID, KID and LPIPS (e.g., CelebA-HQ FID 16.48 vs. 35.27 for MONet and 62.64 for COMET).
- On Falcor3D disentanglement, the 64-dimensional variant reaches MIG 26.45 and MCC 80.42, above COMET's 19.63 and 76.55.
- Predicting x0 with multiple components outperforms epsilon prediction and single-component ablations on CelebA-HQ and CLEVR (CLEVR MSE 6.178 vs 26.094 single-component).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use latent dimension 64 instead of 32 or 128 (ablation) | Decomp Diffusion · Falcor3D | MIG: 11.72±0.05; 12.97±0.02; 26.45±0.16 | Helped |
| Apply PCA to project dimension 128 to 64 (ablation) | Decomp Diffusion · Falcor3D | MIG: 12.97±0.02 -> 16.57±0.02 | Helped |
| Apply PCA to project dimension 128 to 64 (ablation) | Decomp Diffusion · Falcor3D | MCC: 80.27± 0.17 -> 71.19±0.15 | Hurt |
| Predict x0 instead of noise (ablation) | Decomp Diffusion · CelebA-HQ | LPIPS: 0.603 -> 0.089 | Helped |
| Predict x0 instead of noise (ablation) | Decomp Diffusion · CLEVR | LPIPS: 0.3061 -> 0.0122 | Helped |
| Use multiple components instead of one (ablation) | Decomp Diffusion · CelebA-HQ | LPIPS: 0.192 -> 0.089 | Helped |
| Use multiple components instead of one (ablation) | Decomp Diffusion · CLEVR | LPIPS: 0.2236 -> 0.0122 | Helped |
| Use pretrained Stable Diffusion without fine-tuning (ablation) | Decomp Diffusion · CLEVR and KITTI | factor quality | Hurt |
Surprising findings
- Applying the approach to pretrained Stable Diffusion without fine-tuning did not find meaningful factors on CLEVR and KITTI.
- Learning a single component reconstruction underperforms multi-component, but the one-shot Liu et al. textual-inversion baseline failed to produce training-like objects, suggesting pretrained priors are not always faithful for factor learning.
Limitations
- Number of components K must be specified by the user; too large yields duplicate components and too small yields subsets.
- Discovered factors are not guaranteed to be distinct from the original image or each other; large latent dimension can let each factor capture the whole image.
- Computational cost scales with K, and factor-level sampling needs multiple forward passes.
Experimental setup
Datasets: CelebA-HQ Falcor3D Virtual KITTI 2 CLEVR CLEVR Toy Tetris KITTI Anime Van Gogh paintings
Benchmarks: CelebA-HQ Falcor3D Virtual KITTI 2 CLEVR
Metrics: FID KID LPIPS MIG MCC MSE
Baselines: COMET beta-VAE MONet Slot Attention GENESIS-V2 InfoGAN Hessian Penalty
Builds on: COMET DDPM Composable diffusion models Stable Diffusion DDIM
Closest papers by meaning
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.87)
- Compositional Scene Understanding through Inverse Generative Modeling (ICML 2025 · similarity 0.87)
- Interpretable Diffusion via Information Decomposition (ICLR 2024 · similarity 0.87)
- How Diffusion Models Learn to Factorize and Compose (NeurIPS 2024 · similarity 0.87)
- Diffusing Differentiable Representations (NeurIPS 2024 · similarity 0.87)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.