Research map / Image generation / Fast diffusion sampling
CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a pre-trained text-to-image latent diffusion model directly into a conditional few-step (1-4) generator, using a conditional consistency loss and ControlNet-style adapters (with a parameter-efficient variant, PE-CoDi).
Problem
Diffusion models need many sampling steps, and adding image conditions to distilled models is usually done by fine-tuning a distilled model or distilling an already fine-tuned one, which hurts quality and flexibility.
Approach
Adapt the unconditional pre-trained LDM with a duplicated conditional encoder (zero-initialized gated fusion) and distill it in a single stage into a conditional consistency model whose targets are sampled from the adapted model's own PF-ODE, combining a signal-space conditional guidance loss with a noise-space consistency loss.
Key results
- On DF2K super-resolution at 4 steps, CoDi reaches FID 19.637 and LPIPS 0.2656, versus 27.810 for CM-II and 27.806 for GD-I, and beats the 50-step LDM (FID 19.231 is close but CoDi is better than the 20-step LDM at 20.510 only at a few steps).
- PE-CoDi (frozen backbone, 364M trainable params) achieves FID 25.214 at 4 steps on super-resolution, outperforming the ControlNet baseline at 250 steps.
- Single-step InstructPix2Pix-style editing from CoDi is reported as visually comparable to 200-step IP2P, with no quantitative numbers given for editing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill with CoDi rather than GD-II | LDM for super-resolution · Super-resolution (DF2K) | FID: 23.675 -> 19.637 | Helped |
| Distill with CoDi rather than GD-II | LDM for super-resolution · Super-resolution (DF2K) | LPIPS: 0.2796 -> 0.2656 | Helped |
| Use parameter-efficient CoDi rather than GD-II | LDM for inpainting · Inpainting (ImageNet) | FID: 15.950 -> 14.700 | Helped |
| Use parameter-efficient CoDi rather than ControlNet | LDM for inpainting · Inpainting (ImageNet) | LPIPS: 0.2260 -> 0.2231 | Helped |
| Use parameter-efficient CoDi rather than GD-II | LDM for depth-to-image · Text-guided Depth-to-image (WebLI) | FID: 26.513 -> 23.047 | Helped |
| Use parameter-efficient CoDi rather than GD-II | LDM for depth-to-image · Text-guided Depth-to-image (WebLI) | CLIP: 0.2870 -> 0.2874 | Helped |
| Add distillation PF-ODE (ablation) | CoDi for super-resolution · Super-resolution (DF2K) | FID: 20.307 -> 19.637 | Helped |
| Add distillation PF-ODE (ablation) | CoDi for super-resolution · Super-resolution (DF2K) | LPIPS: 0.2733 -> 0.2656 | Helped |
| Add noise-consistency loss (ablation) | CoDi for super-resolution · Super-resolution (DF2K) | FID: 25.728 -> 19.637 | Helped |
| Add noise-consistency loss (ablation) | CoDi for super-resolution · Super-resolution (DF2K) | LPIPS: 0.3252 -> 0.2656 | Helped |
| Initialize from text-to-image pretraining rather than random initialization (ablation) | CoDi for super-resolution · Super-resolution | FID | Helped |
| Sample one consistent time t across a batch rather than mixed t values (ablation) | CoDi for super-resolution · Super-resolution | FID | Helped |
Surprising findings
- Using the adapted model's own PF-ODE for target sampling, instead of the frozen pretrained model, improves results, and a single shared timestep per batch beats mixed timesteps.
Limitations
- Relies on an internal text-to-image LDM, so results are not directly reproducible with public weights.
- Quantitative comparisons on inpainting are only for reference since resolution differs from Palette.
- Editing results are shown qualitatively only.
Experimental setup
Datasets: ImageNet WebLI text-instructed image editing data (InstructPix2Pix) Real-ESRGAN degradation pipeline (DF2K)
Benchmarks: DF2K real-world super-resolution (3,000 degraded pairs) ImageNet inpainting WebLI depth-to-image
Metrics: FID LPIPS CLIP score
Baselines: ControlNet LDMs (DDIM/DPM-Solver) CM-II GD-II RealESRGAN StableSR DiffIR Palette Repaint InstructPix2Pix
Builds on: Consistency Models Progressive Distillation ControlNet Guided Distillation
Closest papers by meaning
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.90)
- Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion (ICML 2024 · similarity 0.89)
- High-Fidelity Diffusion-Based Image Editing (AAAI 2024 · similarity 0.89)
- Diffusion Self-Distillation for Zero-Shot Customized Image Generation (CVPR 2025 · similarity 0.89)
- DCI: Dual-Conditional Inversion for Boosting Diffusion-Based Image Editing (NeurIPS 2025 · similarity 0.89)
- Random Conditioning for Diffusion Model Compression with Distillation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.