Research map / Image generation / Fast diffusion sampling
Distilling ODE Solvers of Diffusion Models into Smaller Steps
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-ODE solvers, which add a single learnable parameter per timestep to existing diffusion ODE solvers and distill a teacher's multi-step trajectory into fewer student steps using one batch, avoiding network fine-tuning.
Problem
Diffusion model sampling needs many function evaluations; learning-free ODE solvers lose fidelity at small NFE, while learning-based distillation requires costly retraining of the denoising network.
Approach
Linearly combine the current denoising output with the difference to the next-timestep output using one learned scalar per timestep, keeping the denoising network frozen; the scalars are fit by distilling a teacher run with C times more steps on a single batch and reused for other batches.
Key results
- D-DDIM with 25 NFE produces FID comparable to DDIM with 250 NFE on the data prediction setup, roughly a 10x speedup.
- D-ODE solvers improve FID over DDIM, iPNDM, DPM-Solver3 and DEIS3 especially at small NFE, with D-DPM-Solver3 surpassing DPM-Solver3 by a significant margin at extremely small NFE.
- D-EDM reaches 3 FID on CIFAR-10 in 2.55 time units versus 187.25 for consistency distillation and 106.16 for progressive distillation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize ODE solver parameters instead of the denoising network | D-EDM versus CD and PD · CIFAR-10 | computational time to achieve 3 FID and generate 50k samples: 2.55 vs 187.25; 2.55 vs 106.16 | Helped |
| Apply the D-ODE solver formulation to existing ODE solvers | Diffusion model sampling · Image generation benchmarks | FID | Helped |
| Use D-ODE solvers at larger NFE | Diffusion model sampling · Image generation benchmarks | FID | No clear effect |
| Use D-ODE solvers at extremely small NFE | DDIM sampling · Image generation benchmarks | FID | Mixed |
Surprising findings
- At around 2 NFE, DDIM occasionally slightly outperforms D-DDIM, which the authors attribute to 2-step DDIM lacking capacity to distill knowledge from the teacher.
Limitations
- Single scalar parameter per timestep may be insufficient for high-resolution image generation, as the authors note.
- Improvement becomes marginal at large NFE, converging to the teacher's FID.
- DDIM teacher is used as default, since multi-step solvers make intermediate targets hard to define.
Experimental setup
Datasets: CIFAR-10 CelebA ImageNet FFHQ LSUN bedroom
Benchmarks: CIFAR-10 CelebA ImageNet FFHQ LSUN bedroom
Metrics: FID computational time to reach 3 FID
Baselines: DDIM iPNDM DPM-Solver DEIS EDM consistency distillation progressive distillation
Builds on: DDIM DPM-Solver DEIS EDM PNDM
Closest papers by meaning
- On Accelerating Diffusion-Based Sampling Processes via Improved Integration Approximation (ICLR 2024 · similarity 0.92)
- S4S: Solving for a Fast Diffusion Model Solver (ICML 2025 · similarity 0.92)
- Adaptive Stochastic Coefficients for Accelerating Diffusion Sampling (NeurIPS 2025 · similarity 0.91)
- Simple and Fast Distillation of Diffusion Models (NeurIPS 2024 · similarity 0.91)
- A Unified Sampling Framework for Solver Searching of Diffusion Probabilistic Models (ICLR 2024 · similarity 0.91)
- Fast ODE-based Sampling for Diffusion Models in Around 5 Steps (CVPR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.