Research map / Image generation / Fast diffusion sampling
Diffusion Model is Effectively Its Own Teacher
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self Step-Distillation (SSD and implicit iSSD), where a diffusion model is distilled from its own later-step predictions/features to improve its own sample quality without external teacher or data.
Problem
Diffusion models are typically distilled from external teachers or for speed; it is unexplored whether a diffusion model can self-distill to improve its own generation quality beyond the original model at the same step count.
Approach
Fuse predictions (SSD) or intermediate layer activations (iSSD) at adjacent timesteps t and s from the same frozen model to form a teacher, and train a student with a fused computation flow to match it (N-step to N-step self-distillation); the fusion is removed at inference, and the approach is linked to high-order DPM-Solver.
Key results
- iSSD on DiT-XL/2 with 50 NFEs achieves FID 2.02 on ImageNet 256x256, beating the original 250-step DiT-XL/2 (FID 2.14) under the same ODE sampler.
- Applying iSSD to a 32-step progressively distilled DiT-XL/2 yields FID 1.99, versus 2.24 for the distilled model alone.
- On Stable Diffusion v1.5, SSD with 10 NFEs surpasses DDIM/DPM-Solver-2/PLMS with 20 NFEs on COCO FID (e.g., 10.94 vs 12.68 for DDIM at 20 NFEs on COCO2014).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply iSSD to DiT-XL/2 | DiT-XL/2 · ImageNet 256×256, 50 steps | FID: 2.26 -> 2.02 | Helped |
| Apply explicit SSD to DiT-XL/2 | DiT-XL/2 · ImageNet 256×256, 50 steps | FID: 2.26 -> 2.21 | Helped |
| Apply iSSD to DiT-XL/2 | DiT-XL/2 · ImageNet 256×256, 10 steps | FID: 12.38 -> 4.92 | Helped |
| Apply iSSD to DiT-XL/2 | DiT-XL/2 · ImageNet 512×512, 50 steps | FID: 3.28 -> 2.86 | Helped |
| Apply iSSD to DiT-S/2, DiT-B/2, and DiT-L/2 | DiT models · ImageNet 256×256, 50 steps | FID: 29.21 -> 27.75; 12.84 -> 11.77; 4.77 -> 4.41 | Helped |
| Apply Self Step-Distillation to Stable Diffusion v1.5 | Stable Diffusion v1.5 · COCO2014 validation, 10 NFEs | FID: 12.78 -> 10.94 | Helped |
| Apply Self Step-Distillation to Stable Diffusion v1.5 | Stable Diffusion v1.5 · COCO2017 validation, 10 NFEs | FID: 22.79 -> 20.85 | Helped |
| Apply iSSD to progressively distilled DiT-XL/2 | Progressive Distillation · ImageNet 256×256, 32 steps | FID: 2.24 -> 1.99 | Helped |
| Apply iSSD to progressively distilled DiT-XL/2 | Progressive Distillation · ImageNet 256×256, 4 steps | FID: 8.75 -> 6.10 | Helped |
| Use iSSD rather than explicit SSD (ablation) | DiT-XL/2 · ImageNet 256×256, 10 steps | FID: 7.99 -> 4.92 | Helped |
| Use Eq.10 training objective with an ε-prediction DiT-XL/2 (ablation) | DiT-XL/2 · ImageNet 256×256, 50 steps | FID: 2.26 -> 2.81 | Hurt |
| Use Eq.12 instead of Eq.10 with an ε-prediction DiT-XL/2 (ablation) | DiT-XL/2 · ImageNet 256×256, 50 steps | FID: 2.26 -> 2.21 | Helped |
Surprising findings
- Self-distillation with synthetic data generated by the model itself yields gains close to those with real data (FID 5.02-5.07 vs 4.92 for 10-step DiT-XL/2).
- Applying self-distillation to a single late denoising step gives the largest quality benefit.
Limitations
- Approximation of the intermediate-step prediction remains suboptimal.
- Method is relatively sensitive to hyperparameters such as the fusion weight lambda.
- Self-distilled 10-step result is slightly worse than directly distilled progressive-distillation model (4.19 vs 4.92 FID at 10 steps).
Experimental setup
Datasets: ImageNet COCO2014 COCO2017
Benchmarks: ImageNet 256x256 ImageNet 512x512 COCO2014 COCO2017
Metrics: FID sFID IS Precision Recall
Baselines: DDIM DPM-Solver-2 PLMS Progressive Distillation Shortcut Model SiT-XL/2 REPA
Builds on: DiT DDIM DPM-Solver Progressive Distillation Stable Diffusion
Closest papers by meaning
- One-step Diffusion with Distribution Matching Distillation (CVPR 2024 · similarity 0.92)
- Simple and Fast Distillation of Diffusion Models (NeurIPS 2024 · similarity 0.92)
- Simple Distillation for One-Step Diffusion Models (NeurIPS 2025 · similarity 0.92)
- A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models (ICML 2024 · similarity 0.91)
- Multistep Distillation of Diffusion Models via Moment Matching (NeurIPS 2024 · similarity 0.91)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.