AI Paper Atlas

Research map / Image generation / Text-to-image diffusion

Adding Conditional Control to Text-to-Image Diffusion Models

Lvmin Zhang, Anyi Rao, Maneesh Agrawala

ICCV 2023 · Image generation › Text-to-image diffusion · approach: Compositional text-to-image guidance · 6,074 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace zero convolutions with Gaussian-initialized convolutions (i.e. add zero convolution) (ablation)ControlNet · sketch, no/insufficient promptsqualitative image quality: performance drops to about the same as ControlNet-liteImproved
add a full trainable copy of encoder blocks instead of a single convolution layer (ablation)ControlNet · sketch, no/insufficient promptsqualitative image quality: ControlNet-lite fails with no or insufficient promptsImproved
use full ControlNet instead of ControlNet-lite (ablation)ControlNet · Sketch user studyresult quality (average ranking): 3.93 -> 4.22Improved
use full ControlNet instead of ControlNet-lite (ablation)ControlNet · Sketch user studycondition fidelity (average ranking): 4.09 -> 4.28Improved
use full ControlNet instead of ControlNet-lite (ablation)ControlNet · ADE20K reconstructionIoU: 0.32 -> 0.35Improved
use full ControlNet instead of ControlNet-lite (ablation)ControlNet · ADE20K segmentation generationFID: 17.92 -> 15.27Improved
ControlNet vs retraining a depth model from scratch on a large cluster (SDv2-D2I)ControlNet · depth-to-imageuser identification precision: 0.52 ± 0.17No clear effect
add ControlNet segmentation conditioning to Stable Diffusion vs PITIControlNet · ADE20K segmentation generationFID: 19.74 -> 15.27Improved
add ControlNet instead of VQGAN baseline trained from scratchControlNet · ADE20K segmentation generationFID: 26.28 -> 15.27Improved
increase Sketch-Guided Diffusion edge guidance scale from 1.6 to 3.2 (ablation)Sketch-Guided Diffusion · Sketch user studyresult quality vs fidelity ranking: quality 3.21 -> 2.52, fidelity 2.31 -> 3.28Mixed
train on limited 1k images (ablation)ControlNet · dataset-size robustnessqualitative: training does not collapse with 1k imagesNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy