Research map / Image generation / Text-to-image diffusion
Simplifying Control Mechanism in Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Simple-ControlNet, which adds controllability to Stable Diffusion via a single-scale 8-layer conditional projection plus LoRA on self-attention, cutting control parameters from ~344.5M to 2.7M.
Problem
ControlNet adds parameters nearly half the size of the base diffusion U-Net (about 344.5M), making control mechanisms heavy to train and deploy.
Approach
Replace ControlNet's multiscale condition encoder with a single-scale shallow convolutional projection added to the U-Net's top-level features, and use LoRA on self-attention layers to learn the condition information embedded in hidden states.
Key results
- Simple-ControlNet uses 2.7M control parameters versus 344.5M for ControlNet.
- Without CFG, Simple-ControlNet achieves Depth FID 15.59 vs ControlNet 21.28 and Canny FID 12.93 vs 26.58 on COCO val2014.
- Human study shows Simple-ControlNet leads in image fidelity and slight advantages in alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ControlNet with Simple-ControlNet | Stable Diffusion v1-5 · Depth, HED Boundary, and Canny Edge without CFG | FID: Depth: 21.28 -> 15.59; HED Boundary: 41.75 -> 10.11; Canny Edge: 26.58 -> 12.93 | Helped |
| Replace ControlNet with Simple-ControlNet | Stable Diffusion v1-5 · Depth, HED Boundary, and Canny Edge without CFG | CLIP Score: Depth: 0.2895 -> 0.2949; HED Boundary: 0.2823 -> 0.3025; Canny Edge: 0.2879 -> 0.3065 | Helped |
| Replace ControlNet with Simple-ControlNet | Stable Diffusion v1-5 · Depth, HED Boundary, and Canny Edge without CFG | condition consistency (RMS/SSIM): Depth RMS: 0.1372 -> 0.1326; HED RMS: 0.1942 -> 0.1384; Canny SSIM: 0.4333 -> 0.5758 | Helped |
| Replace ControlNet with Simple-ControlNet | Stable Diffusion v1-5 · Depth, HED Boundary, and Canny Edge with CFG = 7.5 | FID: Depth: 11.86 -> 10.10; HED Boundary: 11.92 -> 7.97; Canny Edge: 10.63 -> 8.71 | Helped |
| Replace ControlNet with Simple-ControlNet | Stable Diffusion v1-5 · Depth, HED Boundary, and Canny Edge with CFG = 7.5 | CLIP Score: Depth: 0.3148 -> 0.3147; HED Boundary: 0.3147 -> 0.3133; Canny Edge: 0.3140 -> 0.3139 | Mixed |
| Replace ControlNet with Simple-ControlNet | Stable Diffusion v1-5 · Depth, HED Boundary, and Canny Edge with CFG = 7.5 | condition consistency (RMS/SSIM): Depth RMS: 0.1342 -> 0.1317; HED RMS: 0.1347 -> 0.1344; Canny SSIM: 0.4631 -> 0.5740 | Helped |
| Deactivate the projection layer while retaining LoRA (λP = 0, λL = 1) (ablation) | Simple-ControlNet · Condition-controlled generation (Figure 3) | condition relevance | Hurt |
| Add the projection layer while retaining LoRA (λP = 1, λL = 1) (ablation) | Simple-ControlNet · Condition-controlled generation (Figure 3) | condition relevance | Helped |
| Add LoRA while retaining the projection layer (λP = 1, λL = 1) (ablation) | Simple-ControlNet · Condition-controlled generation (Figure 3) | condition relevance | Helped |
| Add the projection layer without LoRA (λP = 1, λL = 0) (ablation) | Simple-ControlNet · Condition-controlled generation (Figure 3) | output variation | No clear effect |
Surprising findings
- Projection layer alone produces little variation in outputs; the LoRA layer is what makes generations follow the condition.
Limitations
- Results on Depth and HED/Canny reported only; evaluation limited to three conditions in quantitative comparison.
- Projection layer alone cannot carry rich semantics, so LoRA is required.
Experimental setup
Datasets: COYO-700M COCO val2014
Benchmarks: COCO val2014
Metrics: FID CLIP score RMS SSIM human preference rate
Baselines: ControlNet ControlNet v1.1 T2I-Adapter Uni-ControlNet
Builds on: ControlNet LoRA Stable Diffusion DDPM
Closest papers by meaning
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.92)
- FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation (ICML 2025 · similarity 0.90)
- Universal Few-shot Spatial Control for Diffusion Models (NeurIPS 2025 · similarity 0.89)
- FreeControl: Efficient, Training-Free Structural Control via One-Step Attention Extraction (NeurIPS 2025 · similarity 0.89)
- Local Conditional Controlling for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- EMControl: Adding Conditional Control to Text-to-Image Diffusion Models via Expectation-Maximization (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.