Research map / Image generation / Fast diffusion sampling
AdaDiff: Adaptive Step Selection for Fast Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaDiff, a lightweight policy-gradient-trained step selection network that picks a per-prompt number of DDIM denoising steps for Stable Diffusion and ModelScopeT2V, cutting inference time by 33-40% at similar quality.
Problem
Diffusion models use a fixed, one-size-fits-all number of denoising steps for all prompts, wasting compute on simple prompts while complex prompts may need more steps.
Approach
Learn an instance-specific step usage policy from the prompt text via a lightweight selection network trained with REINFORCE-style policy gradient on a reward balancing step savings and an IQS image-quality score, with a relative top-k criterion for "high quality".
Key results
- On COCO 2017 with SD-v2.1, AdaDiff averages 28.61 steps and reduces time by 39.7% vs fixed 50 steps with similar IQS (0.412 vs 0.419) and FID (21.92 vs 22.13).
- On MSR-VTT with ModelScopeT2V, AdaDiff reduces generation time by 35.8% (13.6s vs 21.2s) while improving IQS over fixed 50 steps (-0.532 vs -0.518 is slightly lower; FID 45.02 vs 44.85).
- Applied to SDXL-Turbo, AdaDiff achieves a 52.7% speed-up over fixed 5 steps with IQS 0.791 vs 0.801.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdaDiff's adaptive step policy instead of fixed 50-step sampling | SD-v2.1 · COCO-2017 | time: 2.24 -> 1.35; ↑39.7% | Helped |
| Use AdaDiff's adaptive step policy instead of fixed 50-step sampling | ModelScope · MSR-VTT | time per video: 21.2 -> 13.6; ↑35.8% | Helped |
| Use AdaDiff's adaptive step policy instead of fixed 50-step sampling | SDXL-Euler · COCO-2017 5k validation | time: 5.59 -> 3.31; ↑40.8% | Helped |
| Use AdaDiff on top of SDXL-Turbo | SDXL-Turbo · COCO-2017 | time: 0.55 -> 0.26; ↑52.7% | Helped |
| Use AdaDiff on top of SSD-1B | SSD-1B · COCO-2017 | time: 3.60 -> 2.12; ↑41.1% | Helped |
| Transfer AdaDiff's COCO-trained policy to Laion-COCO | SD-v2.1 · Laion-COCO (zero-shot) | time: 2.27 -> 1.38; ↑39.2% | Helped |
| Transfer AdaDiff's InternVid-trained policy to MSR-VTT | ModelScope · MSR-VTT (zero-shot) | time per video: 21.20 -> 14.03; ↑33.8% | Helped |
| Remove the image-quality reward and optimize step savings only (ablation) | AdaDiff · COCO | IQS: 0.412 -> 0.335 | Hurt |
| Replace the IQS reward with CLIP (ablation) | AdaDiff · COCO | IQS: 0.412 -> 0.334 | Hurt |
| Replace the IQS reward with NIQE (ablation) | AdaDiff · COCO | IQS: 0.412 -> 0.392 | Hurt |
| Use an absolute IQS threshold instead of relative top-k ranking (ablation) | AdaDiff · COCO | IQS: 0.412 -> 0.377 | Hurt |
| Use smaller top-k or larger image-reward weight λ (ablation) | AdaDiff · Image generation | average denoising steps | Mixed |
Surprising findings
- The learned policy transfers zero-shot across datasets (COCO to Laion-COCO, InternVid to MSR-VTT) with 33.8-39.2% time savings.
- Standard CLIP and NIQE metrics are argued to be inadequate, motivating the IQS metric.
Limitations
- Reward relies on an IQS quality model and relative top-k ranking, which requires generating images at all ten candidate step counts during training.
- Quality gains over fixed step baselines are often matched rather than exceeded; AdaDiff uses more average steps than some step-reduction baselines (e.g., SDXL-Lightning).
Experimental setup
Datasets: MS COCO 2017 Laion-COCO DiffusionDB MSR-VTT InternVid
Benchmarks: MS COCO 2017 Laion-COCO DiffusionDB MSR-VTT
Metrics: IQS CLIP score IS FID NIQE average denoising steps generation time
Baselines: SD-v2.1 fixed 50 steps SD-v2.1 fixed 28/31/32 steps Random step policy Heuristic word-count policy Perplexity policy SDXL-DPMSolver SDXL-Lightning SSD-1B DeepCache SDXL-Turbo ModelScopeT2V
Builds on: DDIM Latent Diffusion Models policy gradient (REINFORCE) Stable Diffusion
Closest papers by meaning
- One Step Diffusion via Shortcut Models (ICLR 2025 · similarity 0.89)
- Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation (ICLR 2024 · similarity 0.89)
- Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generation (CVPR 2025 · similarity 0.89)
- David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.89)
- A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models (ICML 2024 · similarity 0.89)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.