Research map / Image generation / Fast diffusion sampling
SparseDiT: Token Sparsification for Efficient Diffusion Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SparseDiT, which fine-tunes pretrained DiT-style models with a tri-segment token-sparsification architecture (poolingformer bottom, sparse-dense token modules middle, dense top) plus a timestep-dependent pruning rate, cutting FLOPs by 43-56% with similar FID.
Problem
Diffusion Transformers are expensive due to quadratic self-attention over tokens and many sampling steps, and existing efficiency methods mostly target sampling steps or generic token reduction rather than DiT-specific architectural inefficiency.
Approach
Replace bottom-layer self-attention with poolingformers (global average pooling of values), use sparse-dense token modules in middle layers so most transformer blocks run on a small set of pooled sparse tokens, and increase token density over denoising timesteps via a piecewise linear pruning-rate schedule.
Key results
- On ImageNet 512x512, SparseDiT-XL reduces FLOPs by 55% (525 to 235 G) and raises throughput by 175% with FID 3.13 vs 3.04 for DiT-XL.
- On video generation, SparseDiT reduces FLOPs by 56% (1894 to 828 G) across four datasets with comparable FVD.
- On PixArt-alpha text-to-image, SparseDiT reduces FLOPs by 38% and improves throughput by 69% while FID drops from 4.53 to 4.29.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SparseDiT token sparsification | DiT-XL · ImageNet-1k 512×512 | Throughput: 0.249 -> 0.685 (+175%) | Helped |
| Apply higher pruning, r∈[0.90, 0.96] | DiT-XL · ImageNet-1k 512×512 | FID: 3.04 -> 3.13 | Hurt |
| Apply SparseDiT token sparsification | PixArt-α · SAM | Throughput: 0.414 -> 0.701 (+69%) | Helped |
| Apply SparseDiT token sparsification | PixArt-α · SAM | FID: 4.53 -> 4.29 | Helped |
| Apply SparseDiT token sparsification | Latte-XL · Video datasets (FaceForensics, SkyTimelapse, UCF101, Taichi-HD) | FVD: FaceForensics: 24.10 -> 24.51; SkyTimelapse: 40.78 -> 39.02; UCF101: 284.54 -> 288.90; Taichi-HD: 89.63 -> 84.13 | Mixed |
| Use 4 SDTMs instead of fewer SDTMs (ablation) | SparseDiT · ImageNet-1k 256×256 | FID: 3.86 -> 2.38; 2.51 -> 2.38 | Helped |
| Use 1 SDTM, yielding a U-shaped architecture (ablation) | SparseDiT · ImageNet-1k 256×256 | training stability | Hurt |
| Use 1 or 2 poolingformers rather than 3 (ablation) | SparseDiT · ImageNet-1k 256×256 | FID: 2.56 -> 2.38 | Helped |
| Use 0 poolingformers (ablation) | SparseDiT · ImageNet-1k 256×256 | training stability | Hurt |
| Dynamically adjust token count across denoising timesteps rather than keep it constant (ablation) | SparseDiT · ImageNet-1k 256×256 | FID: 2.48 -> 2.38; 2.63 -> 2.50 | Helped |
| Combine SparseDiT with 25-step DDIM (ablation) | DiT-XL · ImageNet-1k 256×256 | FID: 2.27 -> 3.31 | Hurt |
| Combine SparseDiT with 5-step Rectified Flow (ablation) | DiT-XL · ImageNet-1k 256×256 | FID: 43.40 -> 43.69 | Hurt |
Surprising findings
- Replacing the first two DiT-XL attention maps with constant matrices without fine-tuning yields nearly identical generated images.
- Training is unstable (NaN loss) when no poolingformers are used or when only one SDTM (U-shaped) is used.
Limitations
- Network structure (number of layers per segment and number of sparse tokens) is manually predefined.
- At 512x512 the FID is slightly higher than DyDiT and the speed-up is measured only on A100 hardware.
Experimental setup
Datasets: ImageNet-1k FaceForensics SkyTimelapse UCF101 Taichi-HD SAM
Benchmarks: ImageNet 256x256 ImageNet 512x512 FaceForensics SkyTimelapse UCF101 Taichi-HD SAM text-to-image
Metrics: FID sFID IS Precision Recall FLOPs throughput FVD
Baselines: DiT-B DiT-XL ToMeSD DyDiT TokenCache DiTFastAttn Latte-XL PixArt-alpha
Builds on: DiT Latte PixArt-alpha ToMeSD DDIM
Closest papers by meaning
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.93)
- HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration (ICML 2025 · similarity 0.90)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.90)
- TinyFusion: Diffusion Transformers Learned Shallow (CVPR 2025 · similarity 0.90)
- FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification (NeurIPS 2024 · similarity 0.90)
- ToMA: Token Merge with Attention for Diffusion Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.