Research map / Image generation / Fast diffusion sampling
Factorized Diffusion Architectures for Unsupervised Image Generation and Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a factorized DDPM U-Net with a K-region softmax mask bottleneck and weight-shared masked decoders so that generation training from scratch also yields unsupervised image segmentation.
Problem
Unsupervised image segmentation typically needs separate models, auxiliary losses, or priors, and diffusion models do not natively output segmentations; the paper asks whether a generative denoising model can learn segmentation as a byproduct of its architecture.
Approach
A U-Net denoiser with a softmax mask generator producing K region masks; decoder is weight-shared and run once per region on the encoder features masked by each region, with outputs summed under the masks. Trained only with the DDPM noise-prediction loss, so segmentation emerges from the architectural bottleneck.
Key results
- Generation FID improves over DDPM on all datasets, e.g., Flower-64 13.33 vs 15.81 and ImageNet-64 6.54 vs 7.02.
- Zero-shot VOC 2012 segmentation with ImageNet-trained model reports average accuracy 0.78 and mIOU 0.54 across 20 classes.
- Real-image segmentation requires only one forward (denoising) pass after noising to t=30, and the method outperforms GrabCut, ReDO, and IEM on Flower and CUB per the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the factorized architecture instead of standard DDPM | DDPM image generation on Flower-64 · Flower-64 | FID: 15.81 -> 13.33 | Helped |
| Use the factorized architecture instead of standard DDPM | DDPM image generation on Flower-128 · Flower-128 | FID: 14.62 -> 11.50 | Helped |
| Use the factorized architecture instead of standard DDPM | DDPM image generation on CUB-64 · CUB-64 | FID: 14.45 -> 10.91 | Helped |
| Use the factorized architecture instead of standard DDPM | DDPM image generation on CUB-128 · CUB-128 | FID: 14.01 -> 10.28 | Helped |
| Use the factorized architecture instead of standard DDPM | DDPM image generation on FFHQ-64 · FFHQ-64 | FID: 13.72 -> 12.02 | Helped |
| Use the factorized architecture instead of standard DDPM | DDPM image generation on FFHQ-128 · FFHQ-128 | FID: 13.35 -> 10.79 | Helped |
| Use the factorized architecture instead of standard DDPM | DDPM image generation on ImageNet-64 · ImageNet-64 | FID: 7.02 -> 6.54 | Helped |
| Use parallel weight-shared decoder branches masked by individual segments (ablation) | Factorized diffusion model · CUB | visual feature quality | Helped |
| Replace the parallel masked branches with a single decoder branch concatenating features and masks (ablation) | Factorized diffusion model · CUB | visual feature quality | Hurt |
| Mask the middle features instead of the encoder features (ablation) | Factorized diffusion model · CUB | visual feature quality | Hurt |
| Train decoder branches separately without weight sharing (ablation) | Factorized diffusion model · CUB | visual feature quality | Hurt |
Surprising findings
- Factorized weight-shared masked decoding improves generation FID over the standard DDPM U-Net, not just segmentation.
- Zero-shot transfer from an ImageNet-trained model yields VOC segmentation without any VOC training data.
Limitations
- Mask quality for generated images is evaluated only indirectly via a supervised U-Net, since no ground-truth masks exist.
- Segmentation is evaluated on a fixed small number of segments (K=3) for faces and K regions generally, with class mapping needed for zero-shot VOC transfer.
Experimental setup
Datasets: Flower CUB FFHQ CelebAMask-HQ ImageNet Pixel-ImageNet PASCAL VOC 2012 DAVIS-2017
Benchmarks: Flower CUB CelebAMask-HQ Pixel-ImageNet PASCAL VOC 2012 DAVIS-2017 FFHQ ImageNet
Metrics: FID pixel accuracy IOU DICE
Baselines: DDPM GrabCut ReDO IEM DatasetDDPM-unsup DiffuMask PerturbGAN
Builds on: DDPM U-Net IEM DatasetDDPM
Closest papers by meaning
- U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models (ICLR 2025 · similarity 0.88)
- Denoising Task Difficulty-based Curriculum for Training Diffusion Models (ICLR 2025 · similarity 0.88)
- Denoising Diffusion Bridge Models (ICLR 2024 · similarity 0.88)
- Unsupervised Region-Based Image Editing of Denoising Diffusion Models (AAAI 2025 · similarity 0.87)
- Generalization in diffusion models arises from geometry-adaptive harmonic representations (ICLR 2024 · similarity 0.87)
- Not All Parameters Matter: Masking Diffusion Models for Enhancing Generation Ability (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.