Research map / Image generation / Fast diffusion sampling
A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpeeD, a plug-and-play diffusion training accelerator that suppresses sampling of low-benefit convergence-area time steps (asymmetric sampling) and up-weights rapid-change steps (change-aware weighting), reporting about 3x speed-up across U-Net and DiT.
Problem
Diffusion model training is computationally expensive; existing time-step reweighting and resampling methods do not address the redundancy of nearly pure-noise, easy-to-learn time steps and the under-sampling of rapid-change steps.
Approach
Analyzes process increments over time steps to split them into acceleration, deceleration, and convergence areas; uses asymmetric sampling to down-sample convergence-area steps and a change-aware weighting derived from the gradient of the increment variance to emphasize rapid-change steps.
Key results
- On MetFaces with DiT-XL/2, SpeeD reaches FID 23.4 at 20K iterations versus 132.7 for the baseline, and it speeds up DiT-XL/2 by about 4x in long-term training.
- SpeeD accelerates Min-SNR and CLTS by 2.7x and 2.6x, and further accelerates MDT by at least 4x and FDM by about 1.6x on CIFAR-10.
- Cross-architecture results on ImageNet-1K show FID improving from 26.74 to 20.63 for DiT and from 45.71 to 37.33 for U-Net.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SpeeD's asymmetric sampling and change-aware weighting | DiT-XL/2 · MetFaces | FID: 36.61 -> 21.13 | Helped |
| add SpeeD's asymmetric sampling and change-aware weighting | U-Net · MetFaces | FID: 46.77 -> 22.88 | Helped |
| add SpeeD's asymmetric sampling and change-aware weighting | DiT-XL/2 · FFHQ | FID: 12.86 -> 9.95 | Helped |
| add SpeeD's asymmetric sampling and change-aware weighting | U-Net · FFHQ | FID: 17.37 -> 16.52 | Helped |
| add SpeeD's asymmetric sampling and change-aware weighting | DiT · ImageNet-1K | FID: 26.74 -> 20.63 | Helped |
| add SpeeD's asymmetric sampling and change-aware weighting | U-Net · ImageNet-1K | FID: 45.71 -> 37.33 | Helped |
| add change-aware weighting (CAW) with uniform sampling (ablation) | SpeeD on U-Net · FFHQ | FID: 17.37 -> 16.75 | Helped |
| replace uniform sampling with asymmetric sampling (ablation) | SpeeD on U-Net · FFHQ | FID: 17.37 -> 15.82 | Helped |
| set suppression intensity k to 10 instead of k=5 (ablation) | SpeeD on U-Net · FFHQ | FID: 14.86 -> 16.97 | Hurt |
| set suppression intensity k to 25 instead of k=5 (ablation) | SpeeD on U-Net · FFHQ | FID: 14.86 -> 25.59 | Hurt |
| set symmetry ceiling λ to 0.8 instead of λ=0.6 (ablation) | SpeeD on U-Net · FFHQ | FID: 14.86 -> 16.83 | Hurt |
| set symmetry ceiling λ to 1.0 instead of λ=0.6 (ablation) | SpeeD on U-Net · FFHQ | FID: 14.86 -> 23.77 | Hurt |
Surprising findings
- Re-weighting methods (P2, Min-SNR) reduce FID slowly early in training, suggesting convergence-area steps contribute little benefit.
- Suppressing convergence-area steps too strongly (suppression intensity k=25) degrades FID substantially, indicating those steps still carry useful signal.
Limitations
- Theoretical analysis is derived for DDPM and linear schedules and approximated bounds; generalization to other settings relies on heuristic arguments.
- Hyperparameters (suppression intensity k, symmetry ceiling lambda, threshold tau) require tuning and the default values were selected on FFHQ.
- Acceleration is measured mainly on FID curves at low resolution and with limited training budgets, far from full convergence used in original DiT training.
Experimental setup
Datasets: MetFaces FFHQ CelebA CIFAR-10 ImageNet-1K MS-COCO
Benchmarks: MetFaces FFHQ ImageNet-1K CIFAR-10 MS-COCO
Metrics: FID IS CLIP score
Baselines: DiT-XL/2 P2 Min-SNR Log-Normal CLTS MDT FDM
Builds on: DDPM DiT EDM Min-SNR P2 weighting CLTS
Closest papers by meaning
- Adaptive Non-Uniform Timestep Sampling for Accelerating Diffusion Model Training (CVPR 2025 · similarity 0.85)
- Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture (CVPR 2024 · similarity 0.84)
- Early-Bird Diffusion: Investigating and Leveraging Timestep-Aware Early-Bird Tickets in Diffusion Models for Efficient Training (CVPR 2025 · similarity 0.84)
- Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Models (ICLR 2025 · similarity 0.83)
- Diffusion Model is Effectively Its Own Teacher (CVPR 2025 · similarity 0.83)
- Multi-Architecture Multi-Expert Diffusion Models (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.