Research map / Image generation / Fast diffusion sampling
Random Conditioning for Diffusion Model Compression with Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes random conditioning, pairing noised images with randomly sampled text conditions during distillation, enabling image-free compression of Stable Diffusion v1.4 into smaller students that generate unseen concepts.
Problem
Knowledge distillation of conditional diffusion models fails to transfer teacher knowledge for conditions not covered by the distillation images, and generating teacher images for all text prompts is prohibitively costly.
Approach
During distillation, noised training images are paired with a randomly sampled text condition with a timestep-dependent probability p(t), so the student learns teacher behavior on conditions for which no paired images exist, enabling image-free distillation.
Key results
- Random conditioning reduces FID from 18.15 to 15.76 (B-Base, with teacher init) and from 18.13 to 15.46 without teacher init vs. no random conditioning on MS-COCO 30K.
- Image-free B-Base (0 real images) reaches FID 14.47, IS 36.50, CLIP 0.2932, approaching teacher SDM-v1.4 (13.05, 36.76, 0.2958) and beating BK-SDM-Base (15.76 FID).
- Without random conditioning, the student fails on animal concepts (unseen FID 37.86); with random conditioning and 24K+20M texts unseen FID drops to 24.71 (CLIP 0.2913).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add random conditioning without teacher initialization (ablation) | B-Base · MS-COCO-30k | FID: 18.13 -> 15.46 | Helped |
| add random conditioning with teacher initialization (ablation) | B-Base · MS-COCO-30k | FID: 18.15 -> 15.76 | Helped |
| add real images to random conditioning and teacher initialization (ablation) | B-Base · MS-COCO-30k | FID: 15.76 -> 15.00 | Helped |
| use random conditioning with 24K animal-related texts but no paired animal images (ablation) | B-Base · unseen animal prompts | FID: 37.86 -> 23.26 | Helped |
| extend random-conditioning texts with 20M LAION prompts (ablation) | B-Base · seen and unseen prompts | IS: 27.86 -> 30.27 | Helped |
| use GPT-generated prompts for random conditioning instead of LAION text (ablation) | B-Base · MS-COCO-30k | FID: 15.76 -> 14.98 | Helped |
| distill without random conditioning (ablation) | B-Base · MS-COCO-30k | FID: 15.76 -> 18.15 | Hurt |
| compress channels to C-Micro rather than use BK-SDM-Small | compressed diffusion model · MS-COCO 30K | FID: 16.98 -> 13.42 | Helped |
| compress channels to C-Micro rather than use BK-SDM-Small | compressed diffusion model · MS-COCO 30K | IS: 31.68 -> 32.64 | Helped |
Surprising findings
- Models trained on 10K generated images with random conditioning outperform models trained on 212K images without it.
- Random conditioning contributes more to score gains than the use of real images (Rows 5 and 6 of Table 1 nearly identical).
Limitations
- Experiments limited to text-to-image Stable Diffusion v1.4 teacher and MS-COCO evaluation.
- Teacher is Stable Diffusion v1.4; effect with stronger teachers such as SDXL not tested.
- Timestep-dependent probability p(t) chosen empirically via exponential function.
Experimental setup
Datasets: LAION-Aesthetics V2 6.5+ (212K) LAION (20M extra text prompts) MS-COCO 30K validation GPT-generated prompts (2.2M)
Benchmarks: MS-COCO 30K
Metrics: FID IS CLIP score
Baselines: BK-SDM-Base BK-SDM-Small BK-SDM-Tiny SDM-v1.4 KOALA
Builds on: BK-SDM Stable Diffusion knowledge distillation
Closest papers by meaning
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.91)
- Data-free Distillation of Diffusion Models with Bootstrapping (ICML 2024 · similarity 0.91)
- Denoising Task Difficulty-based Curriculum for Training Diffusion Models (ICLR 2025 · similarity 0.90)
- DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture (CVPR 2025 · similarity 0.90)
- Your Student is Better Than Expected: Adaptive Teacher-Student Collaboration for Text-Conditional Diffusion Models (CVPR 2024 · similarity 0.89)
- Fast constrained sampling in pre-trained diffusion models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.