Research map / Architectures and efficiency / Attention and transformers
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Redesigns FlashAttention for Hopper GPUs using warp-specialized asynchrony, intra-warpgroup GEMM-softmax overlap, and FP8 with block quantization and incoherent processing.
Problem
Attention on newer GPUs (Hopper H100) achieves low utilization (FlashAttention-2 ~35%) because it does not exploit asynchronous hardware units or low-precision FP8 tensor cores, and FP8 attention suffers accuracy loss from outliers.
Approach
Exploit Hopper asynchrony (TMA and WGMMA) via producer-consumer warp specialization with pingpong scheduling and a 2-stage GEMM-softmax pipeline, and adapt the forward pass to FP8 with in-kernel V transposition, block quantization, and Hadamard-based incoherent processing to reduce error.
Key results
- FP16 forward reaches up to 740 TFLOPs/s (75% utilization) and 1.5-2.0x speedup over FlashAttention-2 on H100.
- FP8 forward reaches close to 1.2 PFLOPs/s on H100.
- FP8 with block quantization and incoherent processing has 2.6x lower RMSE than baseline FP8 attention (9.1e-3 vs 2.4e-2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FlashAttention-3 instead of FlashAttention-2 | H100 attention, FP16 forward pass · H100 forward pass, FP16 | speedup: 1.5-2.0× | Helped |
| Use FlashAttention-3 instead of FlashAttention-2 | H100 attention, FP16 backward pass · H100 backward pass, FP16 | speedup: 1.5-1.75× | Helped |
| Use FlashAttention-3 instead of standard attention | H100 attention, FP16 · H100 attention, FP16 | speedup: 3-16× | Helped |
| Use FlashAttention-3 instead of standard attention | Outlier-feature attention, FP16 · Outlier-feature attention, FP16 | RMSE: 3.2e-4 -> 1.9e-4 | Helped |
| Use block quantization and incoherent processing | FP8 FlashAttention-3 · Outlier-feature attention, FP8 | RMSE: 2.4e-2 -> 9.1e-3 | Helped |
| Add GEMM-softmax pipelining (ablation) | FP16 FlashAttention-3 with warp-specialization · Non-causal FP16 attention, batch 4, sequence length 8448, 16 heads, head dimension 128 | TFLOPs/s: 582 -> 661 | Helped |
| Add warp-specialization (ablation) | FP16 FlashAttention-3 with GEMM-softmax pipelining · Non-causal FP16 attention, batch 4, sequence length 8448, 16 heads, head dimension 128 | TFLOPs/s: 570 -> 661 | Helped |
| Remove block quantization (ablation) | FP8 FlashAttention-3 · Outlier-feature attention, FP8 | RMSE: 9.1e-3 -> 9.3e-3 | Hurt |
| Remove incoherent processing (ablation) | FP8 FlashAttention-3 · Outlier-feature attention, FP8 | RMSE: 9.1e-3 -> 2.4e-2 | Hurt |
| Use 3-stage instead of 2-stage GEMM-softmax pipelining (ablation) | FlashAttention-3 · Forward attention | performance | Hurt |
Surprising findings
- The 3-stage pipelining variant performed worse than the 2-stage version because of register pressure and compiler reordering.
- FP8 FlashAttention-3 is behind cuDNN for causal masking with head dimensions 128 and 256, and lacks a persistent kernel.
Limitations
- FP8 variant lacks persistent kernel and load balancing, hurting small-sequence and causal performance
- Focus is on Hopper GPUs; results not shown on other architectures
- Does not yet optimize for LLM inference or study low-precision effects on large-scale training
Experimental setup
Benchmarks: attention forward/backward speed on H100 SXM5 RMSE vs FP64 reference
Metrics: TFLOPs/s runtime ms RMSE
Baselines: FlashAttention-2 FlashAttention-2 Triton cuDNN 9 attention PyTorch standard attention FP8 per-tensor scaling baseline
Builds on: FlashAttention FlashAttention-2 CUTLASS QuIP
Closest papers by meaning
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (ICLR 2024 · similarity 0.93)
- SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization (ICML 2025 · similarity 0.88)
- FlashMask: Efficient and Rich Mask Extension of FlashAttention (ICLR 2025 · similarity 0.88)
- SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training (NeurIPS 2025 · similarity 0.87)
- Flash3D: Super-scaling Point Transformers through Joint Hardware-Geometry Locality (CVPR 2025 · similarity 0.87)
- SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.