Research map / Image generation / Autoregressive and latent generation
DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformers
Hanling Zhang, Rundong Su, Zhihang Yuan, Pengtao Chen, Mingzhu Shen, Yibo Fan, Shengen Yan, Guohao Dai, Yu Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add output caching to arrow attention in the method set (ablation) | DiTFastAttnV2 (SD3) · MS-COCO 1K, delta=0.4 | attention sparsity: 0.30 -> 0.55 | Improved |
| add output caching to arrow attention in the method set (ablation) | DiTFastAttnV2 (SD3) · MS-COCO 1K, delta=0.4 | LPIPS: 0.275 -> 0.238 | Improved |
| add output caching to arrow attention in the method set (ablation) | DiTFastAttnV2 (SD3) · MS-COCO 1K, delta=0.4 | HPSv2: 0.2943 -> 0.2935 | Hurt |
| add CFG sharing to the method set (ablation) | DiTFastAttnV2 (SD3) · MS-COCO 1K, delta=0.4 | attention sparsity: 0.55 -> 0.54 | No clear effect |
| add residual sharing to the method set (ablation) | DiTFastAttnV2 (SD3) · MS-COCO 1K, delta=0.4 | LPIPS: 0.238 -> 0.196 | Improved |
| add residual sharing to the method set (ablation) | DiTFastAttnV2 (SD3) · MS-COCO 1K, delta=0.4 | HPSv2: 0.2935 -> 0.2906 | Hurt |
| replace sliding window with arrow attention in DiTFastAttn | DiTFastAttn (SD3) · MS-COCO 1K | attention sparsity: 0.32 -> 0.44 | Improved |
| use head-wise DiTFastAttnV2 instead of layer-wise attention caching | Attention Caching (N=2) (SD3) · MS-COCO 1K | LPIPS: 0.323 -> 0.238 | Improved |
| use head-wise DiTFastAttnV2 instead of layer-wise attention caching | Attention Caching (N=2) (SD3) · MS-COCO 1K | attention sparsity: 0.50 -> 0.55 | Improved |
| use single-layer RSE calibration with ILP plan search instead of full-output exhaustive search | DiTFastAttn compression plan search · FLUX 2K plan search | search time: over 10 hours -> 15 minutes | Improved |
| use head-wise arrow attention kernel instead of FlexAttention | attention kernel · 16384 visual tokens, 25% sparsity | speedup vs FlashAttention2: 1.22x -> 1.44x | Improved |
| use head-wise arrow attention kernel instead of FlexAttention | attention kernel · 16384 visual tokens, 75% sparsity | speedup vs FlashAttention2: 3.09x -> 3.55x | Improved |
Closest papers by meaning
- DiTFastAttn: Attention Compression for Diffusion Transformer Models (NeurIPS 2024 · similarity 0.92)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.89)
- Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.89)
- Parameterized Stripe Attention for Efficient Video Generation (NeurIPS 2026 · similarity 0.89)
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.89)
- DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
