Research map / Image generation / Fast diffusion sampling
Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Factor Graph Diffusion Models (FG-DMs) that jointly model images and conditioning maps (segmentation, depth, normal, sketch) by adapting frozen Stable Diffusion with per-factor adapters and an attention distillation loss, enabling sampling-based prompt compliance, editing and data augmentation.
Problem
Text-to-image diffusion models have poor prompt compliance (low object recall, text rendering, spatial constraints), and existing visually conditioned models require user-supplied conditions, while joint diffusion models are trained from scratch and do not scale to complex scenes.
Approach
Decompose the joint distribution of image and conditioning variables into a factor graph of conditional diffusion models, each implemented by a lightweight T2I-Adapter on a frozen Stable Diffusion, with classifier-free-guidance-style null conditions and an attention distillation KL loss from SD teacher to student factors; condition maps are sampled at low resolution first, enabling cheap sampling-
Key results
- Sampling-based prompt compliance with N=10 seeds raises average max object recall on ADE20K prompts to 75%, and FG-DM is 4x faster than SD for the same recall (10 DDIM steps for segmentation).
- FG-DM images have 8% higher recall than SD and 4% higher than Attend-and-Excite in the single-run setting on ADE20K.
- Attention distillation improves segmentation condition fidelity by 5.6 points and image FID by 1.29 points; a user study with 10 evaluators per image favored FG-DM for quality (61.37%) and prompt alignment (57.68%) over SD+CEM for the 51 pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FG-DM sampling-based prompt compliance instead of single-seed sampling | FG-DM on ADE20K validation prompts · ADE20K validation prompts | object recall: 69% -> 75% | Helped |
| use FG-DM instead of SD for single-run sampling | prompt-compliant image generation · ADE20K validation prompts | object recall: 8% | Helped |
| use FG-DM instead of Attend & Excite for single-run sampling | prompt-compliant image generation · ADE20K validation prompts | object recall: 4% | Helped |
| use FG-DM SBPC rather than SD SBPC with batch size 10 | prompt-compliant image generation · ADE20K validation prompts | inference speed: 4x | Helped |
| add attention distillation to the adapted FG-DM (ablation) | FG-DM image synthesis · segmentation and normal map conditions | image quality: 1.29/1.3 | Helped |
| add attention distillation to the adapted FG-DM (ablation) | FG-DM condition synthesis · segmentation and normal maps | condition fidelity: 5.6/2.8 | Helped |
| add 1000 FG-DM synthetic training samples (ablation) | BiSeNet · Face part segmentation, MM-CelebA-HQ | mIoU: 51.1 -> 55.1 | Helped |
| add 1000 FG-DM synthetic training samples (ablation) | HRNetV2-W18 · Face landmark estimation, 300W | NME Common: 3.21 -> 3.12 | Helped |
| increase synthetic augmentation from 1000 to 2000 samples (ablation) | BiSeNet · Face part segmentation, MM-CelebA-HQ | mIoU: 55.1 -> 55.0 | Hurt |
| increase synthetic augmentation from 1000 to 2000 samples (ablation) | HRNetV2-W18 · Face landmark estimation, 300W | NME Common: 3.12 -> 3.18 | Hurt |
| jointly train FG-DM factors instead of training them separately (ablation) | FG-DM · MM-CelebA-HQ, ADE20K, Cityscapes, and COCO | image quality (FID) | Helped |
| use joint inference instead of sequential inference (ablation) | FG-DM · image synthesis | FID | Helped |
Surprising findings
- Off-the-shelf SD autoencoder reconstructs segmentation, depth, sketch and normal maps faithfully when they are encoded as 3-channel RGB images, so SD can be used to generate conditions it was never trained on.
- CLIP score failed to reflect prompt adherence: SD had higher CLIP score than FG-DM despite lower object recall.
Limitations
- Runtime grows with longer factor chains since each factor adds computation.
- Fine-grained manipulations (e.g. changing tree branches) still require considerable user effort.
- Pixel-level real image editing is less robust than semantic-level mask editing and the synthesized image does not preserve the original background.
Experimental setup
Datasets: COCO-WholeBody MM-CelebA-HQ ADE20K Cityscapes COCO 300W
Benchmarks: COCO ADE20K MM-CelebA-HQ Cityscapes
Metrics: FID LPIPS Precision Recall CLIP score object recall mIoU NME
Baselines: Stable Diffusion v1.4 Stable Diffusion v1.5 4M-XL Attend-and-Excite ControlNet Make-a-Scene SpaText SegFormer-B4
Builds on: Stable Diffusion T2I-Adapter ControlNet Attend-and-Excite LEDITS++
Closest papers by meaning
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.89)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
- High-Fidelity Diffusion-Based Image Editing (AAAI 2024 · similarity 0.88)
- PromptFix: You Prompt and We Fix the Photo (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.