Research map / Image generation / Fast diffusion sampling
Multistep Distillation of Diffusion Models via Moment Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills many-step diffusion models into 1-8 step generators by matching conditional expectations of clean data along the sampling trajectory (moment matching), with alternating-auxiliary and parameter-space (instant) variants.
Problem
Diffusion models need hundreds of network evaluations to sample; existing distribution-matching distillation methods work well for one step but do not extend cleanly to few-step sampling.
Approach
Cast distribution matching as generalized method of moments on conditional clean-data expectations E[x|z_s], sampling z_s from the conditional q(z_s|z_t, x~) along the trajectory; this extends one-step score-distillation methods to multiple steps, and a parameter-space variant matches teacher gradients without an auxiliary model.
Key results
- On ImageNet, 8-step distilled models with no guidance outperform their 512-step teacher with optimized guidance at 64x64 and 128x128 resolution, setting new state-of-the-art FID among compared methods at 8+ steps.
- For 1-step sampling, other methods (e.g., those compared) show better FID than this method; authors leave improvement for future work.
- On text-to-image at 512x512, the 8-step alternating-optimization distilled model exceeds the base model on zero-shot FID and CLIP score on MS-COCO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill the teacher with moment matching and sample in 8 steps without classifier-free guidance | Simple Diffusion · ImageNet 64x64 | FID | Helped |
| Distill the teacher with moment matching and sample in 8 steps without classifier-free guidance | Simple Diffusion · ImageNet 128x128 | FID | Helped |
| Use moment matching with alternating optimization rather than the instant 2-batch version (ablation) | Moment Matching Distillation · ImageNet, fewer than 8 sampling steps | FID | Helped |
| Use moment matching with alternating optimization rather than the instant 2-batch version (ablation) | Moment Matching Distillation · ImageNet, 8 or more sampling steps | FID | No clear effect |
| Condition the intermediate sample on the current noisy input, q(z_s | x~, z_t), instead of sampling independently, q(z_s | x~) (ablation) | Moment Matching Distillation · ImageNet, single-class samples | sample diversity | Helped |
| Increase classifier-free guidance (ablation) | Moment Matching Distillation · ImageNet | Inception Score | Helped |
| Increase classifier-free guidance (ablation) | Moment Matching Distillation · ImageNet | CLIP Score | Helped |
| Increase classifier-free guidance (ablation) | Moment Matching Distillation · ImageNet | FID | Hurt |
| Use the alternating optimization version rather than the instant 2-batch version (ablation) | Moment Matching Distillation · MS-COCO | zero-shot FID | Helped |
| Use the alternating optimization version rather than the instant 2-batch version (ablation) | Moment Matching Distillation · MS-COCO | CLIP Score | Helped |
Surprising findings
- The 8-step distilled model outperforms its 512-step teacher, which the authors attribute to prediction inconsistencies across timesteps in the teacher.
- Sampling from the conditional q(z_s|z_t, x~) yields much better sample diversity than sampling from the marginal q(z_s|x~) used in prior score-based methods.
Limitations
- 1-step performance lags some competing methods
- Automated metrics (FID, IS, CLIP) are imperfect; no human evaluation reported
- Alternating optimization is not guaranteed to converge (GAN-like); instant variant relies on teacher at a loss minimum
Experimental setup
Datasets: ImageNet licensed text-image pairs MS-COCO
Benchmarks: ImageNet 64x64 ImageNet 128x128 MS-COCO zero-shot (text-to-image)
Metrics: FID Inception Score CLIP score
Baselines: progressive distillation consistency distillation Diff-Instruct adversarial multistep distillation DMD teacher model (512-step / many-step)
Builds on: Diff-Instruct Variational Score Distillation DMD EDM Simple Diffusion
Closest papers by meaning
- Inductive Moment Matching (ICML 2025 · similarity 0.94)
- One-step Diffusion with Distribution Matching Distillation (CVPR 2024 · similarity 0.92)
- Diffusion Model is Effectively Its Own Teacher (CVPR 2025 · similarity 0.91)
- Simple Distillation for One-Step Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.90)
- A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.