Research map / Architectures and efficiency / Attention and transformers
Fast Attention Over Long Sequences With Dynamic Sparse Flash Attention
Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, François Fleuret
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Reformer chunked LSH attention with exact Hash-sparse kernel | GPT2 on MNIST · MNIST | perplexity: 1.76 -> 1.67 | Improved |
| replace Reformer chunked LSH attention with exact Hash-sparse kernel | GPT2 on enwik8 · enwik8 T=4096 | bits/char: 3.32 -> 2.29 | Improved |
| replace FlashAttention with Hash-sparse attention (16 buckets) at T=8192 | OpenWebText2 LM · OpenWebText2 T=8192 | iteration speed: 1.8× | Improved |
| increase sequence length from 8192 to 16384 for Hash-sparse (ablation) | H-LM · OpenWebText2 | iteration speedup: 1.8× -> 2.3× | Improved |
| use Hash-sparse attention instead of FlashAttention | OpenWebText2 LM · OpenWebText2 | validation perplexity per iteration | No clear effect |
| drop 30% of keys/queries heads randomly (ablation) | D-LM · OpenWebText2 T=8192 | perplexity per iteration | No clear effect |
| drop 30% of keys/queries heads randomly (ablation) | D-LM · OpenWebText2 T=8192 | training speed gain: ×1.9 | Improved |
| increase dropping from 30% to 50% to 70% (ablation) | D-LM · OpenWebText2 T=8192 | training speed gain: ×1.9 -> ×2.6 -> ×3.5 | Improved |
| increase dropping to high sparsity (ablation) | D-LM · OpenWebText2 T=8192 | perplexity per iteration | Hurt |
| more sparsity in dropping (ablation) | D-LM · OpenWebText2 T=8192 | time to reach perplexity | No clear effect |
| naive PyTorch implementation of QK dropping instead of SCFA kernel (ablation) | QK-sparse attention · runtime benchmark | runtime vs FlashAttention: only faster when dropping more than 70% | Hurt |
| Reformer fixed-chunk attention vs exact hash-collision coverage (ablation) | Reformer attention · hash coverage | coverage of collisions | Hurt |
Closest papers by meaning
- FlashMask: Efficient and Rich Mask Extension of FlashAttention (ICLR 2025 · similarity 0.89)
- HyperAttention: Long-context Attention in Near-Linear Time (ICLR 2024 · similarity 0.89)
- Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction (NeurIPS 2025 · similarity 0.89)
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (ICLR 2024 · similarity 0.89)
- AdaSplash: Adaptive Sparse Flash Attention (ICML 2025 · similarity 0.88)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
