Research map / Architectures and efficiency / Attention and transformers
Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers
Yifan Zhou, Zeqi Xiao, Tianyi Wei, Shuai Yang, Xingang Pan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Hierarchical KV Enrichment to single-level Top-K attention (ablation) | Top-K sparse attention (L=1) · FFHQ 128x128 | FID: 28.21 -> 26.09 | Improved |
| add KV Reweighting on top of KV Enrichment (ablation) | Top-K sparse attention (L=1) · FFHQ 128x128 | FID: 26.09 -> 24.18 | Improved |
| add KV Reweighting at L=2 (ablation) | LLSA · FFHQ 128x128 | FID: 25.31 -> 24.37 | Improved |
| extend single-level to two-level hierarchy (ablation) | Top-K attention with KV Enrichment and Reweighting · FFHQ 128x128 | FID / throughput: FID 24.18 -> 24.37; throughput 302.92 -> 436.40 | Mixed |
| use block size 64 instead of 16 in Top-K attention (ablation) | Top-K attention (L=1) · FFHQ 128x128 | FID: 28.21 -> 31.33 (K=8 vs K=2); 25.88 -> 26.52 (K=32 vs K=8) | Hurt |
| increase K of single-level baseline from 8 to 32 (ablation) | Top-K attention (L=1) · FFHQ 128x128 | FID: 28.21 -> 25.88; throughput 483.91 -> 357.95 | Improved |
| use LLSA with K=8 instead of baseline Top-K with K=32 (ablation) | Top-K sparse attention · FFHQ 128x128 | FID: 25.88 -> 24.37 | Improved |
| replace VSA/SLA with LLSA | DiT-S on FFHQ 256x256 · FFHQ 256x256 | FID: 39.98 -> 39.29 vs SLA; 40.69 -> 39.29 vs VSA | Improved |
| replace full attention with LLSA | pixel DiT-S · FFHQ 256x256 | FID: 38.77 -> 39.29; throughput 61.64 -> 375.34 | Mixed |
| replace VSA/SLA with LLSA in PixelFlow at highest resolution stage | PixelFlow · ImageNet-256 | FID: 22.58 -> 20.41 vs SLA | Improved |
| replace mask-based key-value backward with sparse index transpose kernel | LLSA · Efficiency benchmark | backward throughput | Improved |
Closest papers by meaning
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.93)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.92)
- Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.91)
- SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.91)
- PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers (NeurIPS 2026 · similarity 0.91)
- Parameterized Stripe Attention for Efficient Video Generation (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
