Research map / Image generation / Fast diffusion sampling
Dynamic Diffusion Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DyDiT, which fine-tunes pretrained DiT models to dynamically adjust computation across diffusion timesteps (timestep-wise dynamic width) and image patches (spatial-wise dynamic token), cutting DiT-XL FLOPs by 51% with 1.73x speedup and FID 2.07 on ImageNet.
Problem
Diffusion Transformers are computationally expensive because they use a static architecture and uniform computation across all diffusion timesteps and spatial patches, although noise-prediction difficulty varies by timestep and region.
Approach
Use learned timestep-conditioned routers to activate subsets of attention heads and MLP channel groups per timestep (pre-computable, so batch-friendly), plus token-level routers that let easy image patches bypass MLP blocks, trained end-to-end with a FLOPs-constrained loss on top of pretrained DiT weights.
Key results
- DyDiT-XL with lambda=0.5 reaches FID 2.07 on ImageNet 256x256 at 57.88 GFLOPs versus 118.69 for DiT-XL (2.27 FID).
- DyDiT-XL (lambda=0.5) accelerates batched generation by 1.73x on a V100 GPU.
- Versus pruning baselines at similar FLOPs on DiT-S fine-grained datasets, DyDiT-S (lambda=0.5) averages FID 13.98 versus 35.44 for magnitude pruning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dynamic timestep-wise width and spatial-wise token computation | DiT-XL · ImageNet 256×256 | FID: 2.27 -> 2.07 | Helped |
| Add dynamic timestep-wise width and spatial-wise token computation | DiT-XL · ImageNet 256×256 | FLOPs (G): 118.69 -> 57.88 | Helped |
| Add dynamic timestep-wise width and spatial-wise token computation | DiT-XL · ImageNet 250-DDPM | seconds/image: 10.22 -> 5.91 | Helped |
| Add dynamic computation to DiT-S | DiT-S · Food, Artbench, Cars, and Birds | FID: Food 14.56 -> 16.74; Artbench 17.54 -> 21.35; Cars 9.30 -> 10.01; Birds 7.69 -> 7.85 | Mixed |
| Add dynamic computation to U-ViT-S/2 | U-ViT-S/2 · CIFAR-10 | FID: 3.12 -> 3.18 | Hurt |
| Add DeepCache with interval 2 | DyDiT-XL λ=0.5 · ImageNet | seconds/image: 5.91 -> 2.99 | Helped |
| Add DeepCache with interval 2 | DyDiT-XL λ=0.5 · ImageNet | FID: 2.07 -> 2.43 | Hurt |
| Combine TDW and SDT instead of using TDW alone (ablation) | DyDiT-S λ=0.5 · ImageNet | FID: 31.89 -> 28.75 | Helped |
| Add TDW rather than using SDT alone (ablation) | DyDiT-S λ=0.5 · ImageNet | FID: 70.06 -> 28.75 | Helped |
| Replace token-level bypass with image-level layer skipping (ablation) | DyDiT-S λ=0.5 · ImageNet | FID: 28.75 -> 30.95 | Hurt |
| Use learned TDW routers rather than random head and channel selection (ablation) | DyDiT-S λ=0.5 · ImageNet | FID: 124.38 -> 31.89 | Helped |
| Use learned TDW routers rather than a manual timestep schedule (ablation) | DyDiT-S λ=0.5 · ImageNet | FID: 34.08 -> 31.89 | Helped |
Surprising findings
- Token-level bypassing of MLP blocks (SDT) alone degrades FID badly (35.12 average) at lambda=0.5, and ToMe token merging degrades DiT performance, unlike on U-Net models.
- DyDiT-XL at lambda=0.5 slightly outperforms the original DiT-XL in FID, suggesting larger models have more redundancy to remove.
Limitations
- Focused only on image generation; video and controllable generation left for future work.
- Combination with distillation-based samplers (LCM) collapses at lower FLOPs ratios, since the difficulty-based training signal is absent there.
- Performance degrades noticeably at low FLOPs ratios (e.g., DyDiT-S at lambda=0.3 FID 59.28).
Experimental setup
Datasets: ImageNet Food Artbench Cars Birds COCO
Benchmarks: ImageNet 256x256 ImageNet 512x512 Food Artbench Cars Birds COCO
Metrics: FID sFID Inception Score Precision Recall FLOPs seconds per image
Baselines: DiT ADM LDM-4 U-ViT DiffuSSM DiM Diff pruning Magnitude pruning Taylor pruning Random pruning ToMe ASE
Builds on: DiT structural pruning Gumbel-Sigmoid DeepCache ToMe
Closest papers by meaning
- D^2iT: Dynamic Diffusion Transformer for Accurate Image Generation (CVPR 2025 · similarity 0.93)
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.90)
- FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Compute (CVPR 2025 · similarity 0.89)
- Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition (CVPR 2025 · similarity 0.89)
- FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification (NeurIPS 2024 · similarity 0.89)
- Effective Diffusion Transformer Architecture for Image Super-Resolution (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.