Research map / Image generation / Fast diffusion sampling
A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Score Estimation (ASE), an early-exiting scheme that skips time-dependent subsets of blocks in pretrained diffusion transformers (DiT, U-ViT, PixArt-α) and fine-tunes them to cut sampling wall-clock time by about 25-30% with little FID loss.
Problem
Diffusion models are slow to sample because the score estimation network is evaluated at every sampling step, and the compute needed per step may vary over time.
Approach
Use a time-dependent exit schedule that drops more blocks (or the non-linear sub-layers of decoder blocks in U-ViT) near the noise end of the trajectory, then fine-tune the model with EMA and time-weighted loss so the remaining blocks adapt to the altered intermediate features.
Key results
- On ImageNet DiT with DDPM-250, D3-DiT reaches FID 8.647 vs baseline 9.078 at 30.46% acceleration.
- On CelebA U-ViT with EM-1000, D1-U-ViT reaches FID 2.250 vs baseline 2.944 at 21.3% acceleration.
- On PixArt-α with DPM-20, ASE gives FID 12.682 vs 12.483 baseline with 14% wall-clock acceleration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ASE with D3 block dropping instead of the baseline model | DiT XL/2 · ImageNet, DDPM-250 | FID: 9.078 -> 8.647 | Helped |
| Apply ASE with D3 block dropping instead of the baseline model | DiT XL/2 · ImageNet, DDPM-250 | acceleration: 30.46% | Helped |
| Apply ASE with D1 block dropping instead of the baseline model | U-ViT-S/4 · CelebA, EM-1000 | FID: 2.944 -> 2.250 | Helped |
| Use more aggressive D3/D6 dropping schedules rather than D1/D2 | U-ViT-S/4 · CelebA, EM-1000 | FID: 2.250 -> 3.217; 2.944 -> 4.379 | Hurt |
| Use ASE instead of the listed baselines | U-ViT · CelebA, DPM-50 | FID: 2.87 -> 1.92 | Helped |
| Use ASE instead of DeeDiff | U-ViT · CelebA, DPM-50 | acceleration: 45.76% -> 23.39% | Hurt |
| Fine-tune PixArt-α with ASE using 1% of original data | PixArt-α-SAM-256 · MS-COCO valid, DPM-20 | FID: 12.483 -> 12.682 | Hurt |
| Use Schedule-1 instead of Schedule-2 (ablation) | ASE U-ViT · CelebA, DPM-25 | FID: 2.456 -> 2.116 | Helped |
| Use Schedule-1 instead of Schedule-3 (ablation) | ASE U-ViT · CelebA, DPM-50 | FID: 3.128 -> 2.144 | Helped |
| Use Schedule-1 instead of Schedule-4 (ablation) | ASE U-ViT · CelebA, DPM-25 and DPM-50 | FID: 2.966 -> 2.116; 3.253 -> 2.144 | Helped |
| Further-train the baseline model (ablation) | U-ViT · CelebA, DPM-25 and DPM-50 | FID: 3.355 -> 4.262; 3.316 -> 4.028 | Hurt |
| Replace the baseline with multi-experts trained for each interval (ablation) | U-ViT · CelebA, DPM-25 and DPM-50 | FID: 3.355 -> 2.987; 3.316 -> 2.942 | Helped |
Surprising findings
- Fine-tuned ASE models often achieve lower FID than the unmodified baseline (e.g., D3-DiT and D1-U-ViT), which the authors attribute to mitigating negative transfer across time intervals.
- Token merging and block caching give little or negative speedup in some settings, due to overhead from token partitioning and bipartite matching.
Limitations
- Dropping schedules are manually designed rather than learned automatically.
- Acceleration is limited to about 30% for the tested configurations, and larger dropping degrades FID.
- Requires fine-tuning after block removal, unlike training-free baselines.
Experimental setup
Datasets: ImageNet CelebA SAM MS-COCO
Benchmarks: ImageNet 256x256 (DiT) CelebA 64x64 (U-ViT) MS-COCO validation (PixArt-α)
Metrics: FID wall-clock time acceleration percentage
Baselines: DeeDiff Token Merging (ToMe) Block Caching Block Caching with scale-shift (SS)
Builds on: DiT U-ViT DDPM DDIM DPM-Solver
Closest papers by meaning
- Simple and Fast Distillation of Diffusion Models (NeurIPS 2024 · similarity 0.92)
- Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models (ICML 2025 · similarity 0.92)
- Accelerating Diffusion Models with Parallel Sampling: Inference at Sub-Linear Time Complexity (NeurIPS 2024 · similarity 0.92)
- Diffusion Model is Effectively Its Own Teacher (CVPR 2025 · similarity 0.91)
- CacheQuant: Comprehensively Accelerated Diffusion Models (CVPR 2025 · similarity 0.91)
- Accelerating Convergence of Score-Based Diffusion Models, Provably (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.