Research map / Image generation / Text-to-image diffusion
Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a segmentation-based adversarial discriminator and a multistep unrolling objective to ControlNet-style Stable Diffusion training so generated images follow input label maps while keeping text editability.
Problem
Layout-to-image (L2I) diffusion models trade off layout faithfulness against text editability; adapter-based methods ignore layouts while fine-tuning overfits and loses text control.
Approach
Train the diffusion UNet with an extra adversarial loss from a jointly trained UperNet segmentation discriminator that classifies denoised images per pixel against the input layout, and unroll K denoising steps so the alignment is enforced over a sampling time window with gradient accumulation.
Key results
- ALDM reaches FID 51.2 and mIoU 63.9 on Cityscapes, and mIoU 36.0 on ADE20K, with TIFA 0.856 and 0.888 respectively.
- Adding adversarial supervision and multistep unrolling raises T2I-Adapter Cityscapes mIoU from 37.1 to 50.1.
- Using ALDM synthetic data for Cityscapes-to-ACDC domain generalization improves HRNet ACDC mIoU from 41.48 to 53.03 (about 11.6 points).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adversarial supervision and multistep unrolling to ControlNet (ablation) | ControlNet · Cityscapes | mIoU: 55.2 -> 63.9 | Helped |
| Add adversarial supervision and multistep unrolling to T2I-Adapter (ablation) | T2I-Adapter · Cityscapes | mIoU: 37.1 -> 50.1 | Helped |
| Add adversarial supervision and multistep unrolling to OFT (ablation) | OFT · Cityscapes | mIoU: 48.9 -> 58.8 | Helped |
| Add adversarial supervision and multistep unrolling to ControlNet (ablation) | ControlNet · Cityscapes | FID: 57.1 -> 51.2 | Helped |
| Use Segmenter rather than UperNet as the discriminator (ablation) | ControlNet + adversarial supervision · Cityscapes | mIoU: 55.2 -> 59.2 | Helped |
| Use a feature-based discriminator (ablation) | ControlNet + adversarial supervision · Cityscapes | mIoU: 55.2 -> 59.6 | Helped |
| Freeze UperNet instead of jointly training the discriminator (ablation) | ControlNet · ADE20K | FID: 29.6 -> 50.8 | Hurt |
| Increase unrolling steps from K = 0 to K = 3 (ablation) | ALDM · Cityscapes | mIoU: 61.5 -> 62.7 | Helped |
| Increase unrolling steps from K = 3 to K = 6 (ablation) | ALDM · Cityscapes | mIoU: 62.7 -> 64.1 | Helped |
| Increase unrolling steps from K = 6 to K = 9 (ablation) | ALDM · ADE20K | mIoU: 34.5 -> 36.0 | Helped |
| Increase unrolling steps from K = 9 to K = 15 (ablation) | ALDM · Cityscapes | TIFA: 0.856 -> 0.882 | Helped |
| Increase unrolling steps from K = 9 to K = 15 (ablation) | ALDM · ADE20K | TIFA: 0.888 -> 0.825 | Hurt |
Surprising findings
- A frozen pretrained segmenter as guidance yields high mIoU but mode collapse and high FID, so joint adversarial training is needed.
- FreestyleNet, which fine-tunes all of Stable Diffusion, achieves high mIoU but very low TIFA editability on Cityscapes (0.300).
Limitations
- Attribute leakage when editing text attributes of one object, inherited from Stable Diffusion.
- May not align well with label maps when text prompts are heavily edited to rare scenarios.
- Overhead of multistep unrolling grows linearly with K.
Experimental setup
Datasets: ADE20K Cityscapes ACDC
Benchmarks: Cityscapes ADE20K ACDC
Metrics: FID mIoU precision recall TIFA per-class IoU
Baselines: PITI FreestyleNet T2I-Adapter ControlNet OFT Pix2PixHD SPADE OASIS SCGAN CLADE GroupDNet Hendrycks-Weather
Builds on: ControlNet Stable Diffusion OASIS UperNet DDIM
Closest papers by meaning
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.87)
- Structure-Guided Adversarial Training of Diffusion Models (CVPR 2024 · similarity 0.87)
- NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models (CVPR 2024 · similarity 0.87)
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.87)
- DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling (CVPR 2025 · similarity 0.87)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.