Research map / Architectures and efficiency / Attention and transformers
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlashAttention-2, an IO-aware exact attention kernel with better work partitioning (fewer non-matmul FLOPs, sequence-length parallelism, warp-level partitioning) that speeds up attention on A100 GPUs.
Problem
Attention runtime and memory scale quadratically with sequence length, and the original FlashAttention still reaches only a fraction of peak GPU FLOPs/s due to suboptimal work partitioning across thread blocks and warps.
Approach
Reduces non-matmul FLOPs in the forward and backward passes, parallelizes over the sequence length dimension in addition to batch and heads, and partitions work between warps to cut shared-memory communication.
Key results
- Reports about 2x speedup over the original FlashAttention on A100 GPUs (abstract-level claim, numbers not in the extract).
Limitations
- Only the truncated background section is available, so detailed evaluation results cannot be verified from the text provided.
Experimental setup
Metrics: runtime FLOPs/s memory
Baselines: FlashAttention PyTorch standard attention Megatron-LM xFormers
Builds on: FlashAttention
Closest papers by meaning
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (NeurIPS 2024 · similarity 0.93)
- FlashMask: Efficient and Rich Mask Extension of FlashAttention (ICLR 2025 · similarity 0.90)
- SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization (ICML 2025 · similarity 0.88)
- Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels (NeurIPS 2025 · similarity 0.87)
- SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration (ICLR 2025 · similarity 0.87)
- Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.