Research map / Image generation / Text-to-image diffusion
Adding Conditional Control to Text-to-Image Diffusion Models
Lvmin Zhang, Anyi Rao, Maneesh Agrawala
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace zero convolutions with Gaussian-initialized convolutions (i.e. add zero convolution) (ablation) | ControlNet · sketch, no/insufficient prompts | qualitative image quality: performance drops to about the same as ControlNet-lite | Improved |
| add a full trainable copy of encoder blocks instead of a single convolution layer (ablation) | ControlNet · sketch, no/insufficient prompts | qualitative image quality: ControlNet-lite fails with no or insufficient prompts | Improved |
| use full ControlNet instead of ControlNet-lite (ablation) | ControlNet · Sketch user study | result quality (average ranking): 3.93 -> 4.22 | Improved |
| use full ControlNet instead of ControlNet-lite (ablation) | ControlNet · Sketch user study | condition fidelity (average ranking): 4.09 -> 4.28 | Improved |
| use full ControlNet instead of ControlNet-lite (ablation) | ControlNet · ADE20K reconstruction | IoU: 0.32 -> 0.35 | Improved |
| use full ControlNet instead of ControlNet-lite (ablation) | ControlNet · ADE20K segmentation generation | FID: 17.92 -> 15.27 | Improved |
| ControlNet vs retraining a depth model from scratch on a large cluster (SDv2-D2I) | ControlNet · depth-to-image | user identification precision: 0.52 ± 0.17 | No clear effect |
| add ControlNet segmentation conditioning to Stable Diffusion vs PITI | ControlNet · ADE20K segmentation generation | FID: 19.74 -> 15.27 | Improved |
| add ControlNet instead of VQGAN baseline trained from scratch | ControlNet · ADE20K segmentation generation | FID: 26.28 -> 15.27 | Improved |
| increase Sketch-Guided Diffusion edge guidance scale from 1.6 to 3.2 (ablation) | Sketch-Guided Diffusion · Sketch user study | result quality vs fidelity ranking: quality 3.21 -> 2.52, fidelity 2.31 -> 3.28 | Mixed |
| train on limited 1k images (ablation) | ControlNet · dataset-size robustness | qualitative: training does not collapse with 1k images | No clear effect |
Closest papers by meaning
- Simplifying Control Mechanism in Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.94)
- Universal Few-shot Spatial Control for Diffusion Models (NeurIPS 2025 · similarity 0.92)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.91)
- Local Conditional Controlling for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.91)
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Readout Guidance: Learning Control from Diffusion Features (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
