Research map / Architectures and efficiency / Attention and transformers
DiTFastAttn: Attention Compression for Diffusion Transformer Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Post-training, training-free compression of diffusion transformer attention via window attention with residual sharing, attention sharing across timesteps, and attention sharing across CFG, with a greedy per-layer per-step plan search.
Problem
Self-attention in diffusion transformers (DiT) has quadratic cost in token length, making high-resolution image and video generation with many denoising steps and CFG expensive, and retraining-based efficient attention designs are costly.
Approach
Identifies three redundancies in DiT attention at inference (spatial locality, similarity of attention outputs across neighboring timesteps, and similarity between conditional and unconditional CFG branches) and exploits them with three training-free techniques: window attention plus cached full-minus-window residual reuse, step-wise attention output reuse, and CFG-wise attention output reuse, sel
Key results
- On PixArt-Sigma at 2048x2048, the D6 setting reduces attention FLOPs to 24% of raw with FID 49.34 vs 51.89 raw, and end-to-end latency drops from 39.86s to 22.27s (about 1.8x).
- On DiT-XL-2 512x512 at D1 to D3, FID stays close to raw (25.32 to 23.76 vs 25.43) with attention FLOPs of 85% to 59%.
- On OpenSora, the configurations D1 through D6 reduce computation by 7.63% to 40.52%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DiTFastAttn at D6 | DiT-XL-2 · ImageNet, 512x512 | IS: 408.16 -> 352.20 | Hurt |
| Apply DiTFastAttn at D6 | DiT-XL-2 · ImageNet, 512x512 | Attn FLOPs: 100% -> 34% | Helped |
| Apply DiTFastAttn at D6 | PixArt-Sigma-XL · MS-COCO, 1024x1024 | FID: 55.65 -> 52.73 | Helped |
| Apply DiTFastAttn at D6 | PixArt-Sigma-XL · MS-COCO, 1024x1024 | CLIP: 31.27 -> 31.18 | Hurt |
| Apply DiTFastAttn at D6 | PixArt-Sigma-XL · MS-COCO, 2048x2048 | Attn FLOPs: 100% -> 24% | Helped |
| Apply DiTFastAttn at D6 | PixArt-Sigma-XL · MS-COCO, 2048x2048 | Latency: 39.86 -> 22.27 | Helped |
| Use ASC only (ablation) | DiT-XL-2 · 512x512 | Attn FLOPs: 50% | Helped |
| Use WA-RS only (ablation) | DiT-XL-2 · 512x512 | Attn FLOPs: 77% | Helped |
| Use WA-RS with ASC (ablation) | DiT-XL-2 · 512x512 | Attn FLOPs: 38% | Helped |
| Use AST only (ablation) | DiT-XL-2 · 512x512 | Attn FLOPs: 0% | Helped |
| Add residual sharing to window attention (ablation) | DiTFastAttn · DiT-XL-2-512 | generation quality | Helped |
| Increase the denoising steps (ablation) | DiTFastAttn · DiT-XL-2-512 | computation compression at maintained quality | Helped |
Surprising findings
- Compressed models at higher compression sometimes obtain lower FID than the raw model on DiT-XL-2 (e.g. D6 FID 16.80 vs raw 25.43), which the authors do not deeply explain.
- Layers and steps show very different redundancy patterns across models, so no universal compression plan transfers between DiT and PixArt-Sigma.
Limitations
- Post-training method cannot benefit from training to avoid performance drop.
- Attention sharing across timesteps stores previous hidden states, increasing VRAM usage.
- Greedy plan search may not find the optimal compression plan; only attention module is accelerated.
Experimental setup
Datasets: ImageNet MS-COCO 2014
Benchmarks: ImageNet MS-COCO 2014
Metrics: FID IS CLIP score attention FLOPs latency
Baselines: Window Attention without residual sharing ASC only AST only WA-RS only WA-RS + ASC
Builds on: FlashAttention-2 DPM-Solver PixArt-Sigma DeepCache PSViT
Closest papers by meaning
- Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.90)
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.90)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.90)
- DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling (NeurIPS 2025 · similarity 0.89)
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.89)
- Parameterized Stripe Attention for Efficient Video Generation (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.