Research map / Image generation / Fast diffusion sampling
Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Score identity Distillation (SiD), a data-free method that distills a pretrained EDM diffusion model into a one-step generator by minimizing a model-based score-matching (Fisher divergence) loss using three score identities.
Problem
Diffusion models need many iterative denoising steps at inference; distilling them into one-step generators typically requires real data, GAN losses, pairwise regression data, or multi-stage procedures.
Approach
Views diffused real and fake distributions as semi-implicit distributions, uses Tweedie's formula and a score projection identity to turn the intractable model-based score-matching loss into a tractable loss, and adds a term (weighted by alpha) that fixes the failure of the naive approximation; trains the generator only on its own synthesized images.
Key results
- SiD reports one-step FIDs of 1.923 (CIFAR-10 unconditional), 1.710 (CIFAR-10 conditional), 1.550 (FFHQ 64x64), 1.628 (AFHQ-v2 64x64) and 1.524 on ImageNet 64x64 (alpha=1.2), surpassing the EDM teacher on all but ImageNet.
- FID decreases roughly exponentially (log-log linear) with the number of processed images; on ImageNet 64x64 with batch size 1024, FID falls below 3 after about 31M images.
- On ImageNet 64x64, SiD (1.524) is exceeded by RIN (1.23 FID, 1000 steps) and VP-EDM teacher (1.36 FID, 511 NFE).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill VP-EDM using single-step SiD | VP-EDM · CIFAR-10, ImageNet 64x64, FFHQ, and AFHQ-v2 | FID | Helped |
| Set α within 0 to 1.2 rather than using a negative α (ablation) | SiD · CIFAR-10 unconditional | FID | Helped |
| Increase α within the tested range, selecting α=1 or α=1.2 (ablation) | SiD · CIFAR-10 unconditional | FID | Helped |
| Set generator Adam β1=0.9 instead of β1=0 (ablation) | SiD generator · FFHQ | FID: more than 0.15 | Helped |
| Set generator Adam β1=0.9 instead of β1=0 (ablation) | SiD generator · Datasets other than FFHQ | FID | No clear effect |
| Set generator-score-network Adam β1=0.9 instead of β1=0 (ablation) | SiD generator score network · All evaluated datasets | convergence | Hurt |
| Increase ImageNet batch size from 1024 to 8192 and reduce learning rate from 5e-6 to 4e-6 (ablation) | SiD · ImageNet 64x64 | FID | Mixed |
| Use batch size 8192 instead of 1024 (ablation) | SiD · ImageNet 64x64 | training convergence speed | Hurt |
| Use the projected score-matching loss rather than the naive score-difference loss (ablation) | SiD · CIFAR-10 | FID | Helped |
| Use the projected score-matching loss alone (α=0), without the fused loss term (ablation) | SiD · CIFAR-10 | FID | Hurt |
| Use the fused SiD loss with score gradients through both score networks | SiD · CIFAR-10, ImageNet 64x64, FFHQ, and AFHQ-v2 | FID | Helped |
| Use the naive loss estimated by reparameterization and Monte Carlo (ablation) | SiD · CIFAR-10 | generation quality | Hurt |
Surprising findings
- A one-step student can exceed its multi-step teacher (EDM) in FID, suggesting reverse-diffusion error accumulation may limit the teacher.
- The naive loss approximation (using only the fake-data score estimator) fails to give meaningful gradients in a toy Gaussian example and in CIFAR-10 experiments.
Limitations
- Requires maintaining three networks (teacher score net, generator score net, generator) of equal size, increasing memory versus Diff-Instruct (about 50-70% more).
- ImageNet 64x64 result does not beat the teacher; training stability issues (FP16 divergence, FID spikes) required batch size 8192 and lower learning rate.
- Hyperparameter alpha chosen empirically (range -0.25 to 1.2) without a finer grid search.
Experimental setup
Datasets: CIFAR-10 ImageNet 64x64 FFHQ 64x64 AFHQ-v2 64x64
Benchmarks: CIFAR-10 32x32 ImageNet 64x64 FFHQ 64x64 AFHQ-v2 64x64
Metrics: FID Inception Score Precision Recall
Baselines: EDM Diff-Instruct DMD CTM Progressive Distillation UFOGen RIN StyleGAN2
Builds on: EDM Diff-Instruct Diffusion-GAN DMD Progressive distillation
Closest papers by meaning
- One-Step Diffusion Distillation through Score Implicit Matching (NeurIPS 2024 · similarity 0.92)
- Distribution Backtracking Builds A Faster Convergence Trajectory for Diffusion Distillation (ICLR 2025 · similarity 0.92)
- Simple and Fast Distillation of Diffusion Models (NeurIPS 2024 · similarity 0.91)
- Diffusion Model is Effectively Its Own Teacher (CVPR 2025 · similarity 0.91)
- One-step Diffusion with Distribution Matching Distillation (CVPR 2024 · similarity 0.91)
- Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instruction (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.