Research map / Image generation / Fast diffusion sampling
Plug-and-Play Diffusion Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills classifier-free guidance and sampling steps into a lightweight external guide network (about 1% of U-Net parameters) while keeping Stable Diffusion v1.5 frozen, so the guide can be plugged into DreamBooth-finetuned variants without retraining.
Problem
Classifier-free guided latent diffusion models need two U-Net passes per denoising step and many steps, making inference slow; standard distillation retrains a full-size student and must be redone for each fine-tuned domain.
Approach
Train an external ControlNet-style guide model that takes the guidance scale as input and injects features into a frozen teacher U-Net to emulate CFG in one forward pass; a tiny variant removes the encoder and the guide transfers across fine-tuned base models without retraining, then progressive halving distills fewer steps.
Key results
- Full guide model with 8 to 16 sampling steps reaches FID comparable to the teacher at moderate guidance (e.g., Full 16 step FID 20.74 at g=8 vs. 21.0 for PLMS 50x2 teacher), while cutting FLOPs by roughly one-third to one-half.
- Tiny guide model (about 1% of parameters, 8.27M) uses 0.51 of teacher FLOPs at 50 steps, but degrades sharply at 8 steps (FID 36.22 at g=8).
- Plugging the trained guide into DreamBooth realistic, 3D cartoon, and watercolor fine-tunes gives CLIP scores close to CFG (e.g., 31.04 vs 32.15 on realistic) with roughly twice the efficiency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace two-pass classifier-free guidance with the tiny guide model in a single pass | Stable Diffusion v1.5 · COCO 30k, g=8 | FLOPs per sampling step: T (338.7) + 7.79 vs 677.5 | Helped |
| Use the tiny guide architecture instead of the full guide architecture (ablation) | Plug-and-Play Diffusion Distillation · COCO 30k, g=8, 8 steps | FID: 29.53 -> 36.22 | Mixed |
| Reduce full-guide sampling from 50 to 8 steps (ablation) | Full guide · COCO 30k, g=8 | FID: 18.19 -> 29.53 | Hurt |
| Reduce tiny-guide sampling from 50 to 8 steps (ablation) | Tiny guide · COCO 30k, g=8 | FID: 19.74 -> 36.22 | Hurt |
| Train the tiny guide with fixed guidance instead of continuous guidance (ablation) | Tiny guide · COCO 30k, 8 steps | CLIP: 30.45 | Helped |
| Apply the pretrained tiny guide to a realistic fine-tuned model without retraining | Stable Diffusion v1.5 · Realistic fine-tune, g=8, 50 steps | FID: 14.71 -> 19.88 | Hurt |
| Apply the pretrained tiny guide to a 3D-cartoon fine-tuned model without retraining | Stable Diffusion v1.5 · 3D Cartoon fine-tune, g=8, 50 steps | FID: 18.00 -> 22.23 | Hurt |
| Apply the pretrained tiny guide to a watercolor fine-tuned model without retraining | Stable Diffusion v1.5 · Watercolor fine-tune, g=8, 50 steps | FID: 38.42 -> 41.87 | Hurt |
| Remove the guide encoder blocks to make the architecture tiny (ablation) | Plug-and-Play Diffusion Distillation · COCO 30k, 8 steps | FID: 29.53 -> 36.22 | Hurt |
Surprising findings
- The tiny guide model's performance degrades drastically at 8 steps, though it matches the full guide model at around 50 steps.
- The distilled student often shows stronger contrast than the CFG teacher on some prompts.
Limitations
- Cannot be run as a simple batch of two like CFG; the U-Net and guide module must run in parallel, increasing GPU memory use.
- Tiny guide model degrades sharply at 8 sampling steps without fixed guidance.
- Evaluation is limited to Stable Diffusion v1.5 and DreamBooth variants on COCO prompts.
Experimental setup
Datasets: LAION COCO
Benchmarks: COCO
Metrics: FID CLIP score FLOPs user study
Baselines: DDIM PLMS Classifier-free guidance teacher Progressive distillation (Meng et al.)
Builds on: classifier-free guidance ControlNet progressive distillation DDIM
Closest papers by meaning
- Elucidating the design space of classifier-guided diffusion generation (ICLR 2024 · similarity 0.92)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.92)
- Diffusion Model is Effectively Its Own Teacher (CVPR 2025 · similarity 0.91)
- A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models (ICML 2024 · similarity 0.91)
- Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation (ICML 2025 · similarity 0.91)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.