Research map / Image generation / Text-to-image diffusion
CCM: Real-Time Controllable Visual Content Creation Using Text-to-Image Consistency Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains ControlNet for text-to-image latent consistency models directly via consistency training (plus a unified multi-condition adapter to transfer DM ControlNets), enabling real-time controllable generation in 1-4 steps.
Problem
Adding new conditional controls (e.g., sketch, depth, pose) to pre-trained text-to-image consistency models for fast controllable generation, since ControlNets trained for diffusion models transfer imperfectly and lose low-level detail and realism.
Approach
Treat the frozen text-to-image consistency model plus a trainable ControlNet as a new conditional consistency model and train the ControlNet from scratch with the consistency training loss; additionally train a unified multi-condition adapter with consistency training to better adapt diffusion-trained ControlNets to CMs.
Key results
- CM's ControlNet trained with consistency training reaches average FID 9.09 at 1 step and 7.61 at 4 steps, close to the DM ControlNet+DM baseline of 7.31 with 50 steps (x2 for CFG).
- Consistency-trained CM ControlNet saves about 100x NFEs and about 26x running time (0.2s per 1024x1024 image at 1 step vs 23.6s for DM).
- Using the unified adapter reduces average FID for DM's ControlNet transferred to CM from 23.19 to 16.96 at 1 step and from 16.16 to 10.27 at 4 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train CM’s ControlNet with consistency training instead of transferring DM’s ControlNet directly | Controllable generation with CM · Sketch2Image, Depth2Image, Mask2Image, 16×SR | FID: 30.71 -> 10.39; 26.08 -> 12.94; 14.67 -> 5.44; 21.32 -> 7.60 | Helped |
| Add a unified adapter to DM’s ControlNet | DM’s ControlNet+CM · Sketch2Image, Depth2Image, Mask2Image, 16×SR | FID: 30.71 -> 20.43; 26.08 -> 19.75; 14.67 -> 13.95; 21.32 -> 13.73 | Helped |
| Train CM’s ControlNet with consistency training rather than use DM’s ControlNet with CM | Controllable generation with CM · Sketch2Image, Depth2Image, Mask2Image, 16×SR | Fidelity: 0.083 -> 0.095; 0.193 -> 0.169; 0.431 -> 0.082; 0.237 -> 0.118 | Mixed |
| Use L1 distance instead of MSE distance (ablation) | CM’s ControlNet consistency training · Sketch2Image | FID: 20.16 -> 9.30 | Helped |
| Use L1 distance instead of Charbonnier distance (ablation) | CM’s ControlNet consistency training · Sketch2Image | FID: 11.91 -> 9.30 | Helped |
| Use L1 distance instead of randomly picking a distance function (ablation) | CM’s ControlNet consistency training · Sketch2Image | FID: 11.37 -> 9.30 | Helped |
| Set consistency-training hyperparameter N to 50 rather than 100, 200, 500, 1000, or use a progressive schedule (ablation) | CM’s ControlNet consistency training · Sketch2Image | FID: 9.68; 10.31; 10.25; 10.37; 10.21 -> 9.30 | Helped |
| Use the general prompt “A high-quality and professional image” rather than paired image captions (ablation) | CM’s ControlNet · Sketch2Image | FID: 10.39 -> 11.25 | Hurt |
Surprising findings
- DM's ControlNet applied to a CM gives high-level semantic control but unrealistic images, and CM's ControlNet transferred to DM still fails at fine detail control.
- The unified adapter trained on five conditions remains effective on an untrained condition (depth), generalizing to Depth2Image.
Limitations
- Consistency-trained ControlNet still shows a small FID gap versus the diffusion baseline at 1 step (9.09 vs 7.31 average), and DM-ControlNet transfer remains sub-optimal.
- Evaluation uses FID and a condition-extractor fidelity metric only; no human evaluation or user study is reported.
- Training data and code are not released; the foundational text-to-image CM training cost (about 160 A100 GPU days) is large.
Experimental setup
Datasets: ImageNet21K WebVision LAION (filtered)
Benchmarks: FID (10k samples) Fidelity on sketch, depth, mask, and 16x super-resolution conditions
Metrics: FID Fidelity (L1 distance between condition extractor output and NFEs running time
Baselines: DM's ControlNet+DM (DDIM 50 steps) DM's ControlNet+CM DM's ControlNet+CM+Adapter
Builds on: Consistency Models ControlNet Latent Consistency Models Stable Diffusion Consistency distillation
Closest papers by meaning
- Phased Consistency Models (NeurIPS 2024 · similarity 0.88)
- Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesis (CVPR 2025 · similarity 0.87)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.87)
- Storybooth: Training-Free Multi-Subject Consistency for Improved Visual Storytelling (ICLR 2025 · similarity 0.86)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.86)
- Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.