Research map / Architectures and efficiency / Attention and transformers
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard attention with IO-aware tiled FlashAttention | BERT-large · BERT-large Wikipedia training | training time: 20.0 ± 1.5 -> 17.4 ± 1.4 minutes | Helped |
| Replace Huggingface attention implementation with FlashAttention | GPT-2 small · OpenWebText | training time: 9.5 days -> 2.7 days (3.5×) | Helped |
| Replace Megatron-LM attention implementation with FlashAttention | GPT-2 medium · OpenWebText | training time: 11.5 days -> 6.9 days (3.0×) | Helped |
| Replace standard attention with FlashAttention | Transformer · LRA average | score: 59.3 -> 59.8 | Helped |
| Replace standard attention with FlashAttention | Transformer · LRA average | speedup: 2.4× | Helped |
| Use block-sparse attention instead of dense FlashAttention (ablation) | Transformer · LRA average | speedup: 2.4× -> 2.8× | Helped |
| Use block-sparse attention instead of dense FlashAttention (ablation) | Transformer · LRA average | score: 59.8 -> 59.6 | No clear effect |
| Increase context length from 1k to 4k (ablation) | GPT-2 small with FlashAttention · OpenWebText | perplexity: 18.2 -> 17.5 | Helped |
| Increase sequence length from 512 to 16384 (ablation) | RoBERTa with FlashAttention · MIMIC-III | micro F1: 52.8 -> 57.1 | Helped |
| Increase sequence length from 512 to 8192 (ablation) | RoBERTa with FlashAttention · ECtHR | micro F1: 72.2 -> 80.7 | Helped |
| Increase sequence length from 512 to 16384 (ablation) | RoBERTa with FlashAttention · MIMIC-III | micro F1: 52.8 -> 50.7 -> 51.7 -> 54.6 -> 56.4 -> 57.1 | Mixed |
| Increase sequence length from 512 to 16384 (ablation) | RoBERTa with FlashAttention · ECtHR | micro F1: 72.2 -> 74.3 -> 77.1 -> 78.6 -> 80.7 -> 79.2 | Mixed |
Closest papers by meaning
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (NeurIPS 2024 · similarity 0.93)
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (ICLR 2024 · similarity 0.90)
- I/O Complexity of Attention, or How Optimal is FlashAttention? (ICML 2024 · similarity 0.89)
- FlashMask: Efficient and Rich Mask Extension of FlashAttention (ICLR 2025 · similarity 0.87)
- Flash3D: Super-scaling Point Transformers through Joint Hardware-Geometry Locality (CVPR 2025 · similarity 0.86)
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.