Research map / Architectures and efficiency / Quantization
SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SageAttention2, a quantized attention kernel that runs Q,K in INT4 (per-thread granularity, with Q smoothing) and P,V in FP8 (with two-level FP32 accumulation), achieving ~3x FlashAttention2 kernel speed on RTX4090 with negligible end-to-end metric loss.
Problem
Attention is quadratic in sequence length and quantizing its matmuls to low precision (INT4 for QK^T, FP8 for PV) causes large accuracy loss due to outliers, INT4's narrow range, and reduced FP8 accumulator precision.
Approach
Smooth Q (in addition to K) by subtracting block means and correcting with a GEMV bias term; per-thread quantization groups tokens according to the mma instruction layout so each thread needs one dequantization scale; quantize P to FP8 with static scale and V per-channel; and use a two-level accumulation (FP22 mma result added into FP32 buffer) to counter the discovered FP22 accumulator.
Key results
- SageAttn2-4b kernel is about 3x faster than FlashAttention2 and 4.5x faster than xformers on RTX4090 (peak 481 TOPS).
- SageAttn2-8b matches FlashAttention3(fp8) speed on Hopper while staying close to full-precision accuracy; FlashAttn3(fp8) degrades CogVideoX and HunyuanVideo metrics.
- SageAttn2-8b gives 1.8x end-to-end speedup on CogVideoX-1.5-5B (1040 s to 577 s) with no metric loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| smooth both Q and K instead of leaving them unsmoothed (ablation) | INT4 Q,K quantization · CogvideoX layers | CosSim: 80.04% -> 99.46% | Helped |
| smooth Q as well as K, rather than smoothing K alone (ablation) | INT4 Q,K quantization · CogvideoX layers | CosSim: 98.07% -> 99.46% | Helped |
| use per-thread rather than per-block quantization (ablation) | INT4 Q,K quantization · CogvideoX layers | Cos Sim: 98.03% -> 99.45% | Helped |
| use per-thread rather than per-token quantization (ablation) | INT4 Q,K quantization · CogvideoX layers | TOPS: 268 -> 283 | Helped |
| use E4M3 rather than E5M2 for P,V (ablation) | SageAttention2-4b · CogvideoX layers | CosSim: 99.20% -> 99.44% | Helped |
| smooth V by subtracting its channel mean (ablation) | SageAttention2-4b · CogvideoX real tensors | Cos Sim: 98.25% -> 99.75% | Helped |
| add per-thread quantization (ablation) | SageAttention2 attention kernel · L20 | TOPS: 284 -> 283 | No clear effect |
| add Q smoothing (ablation) | SageAttention2 attention kernel · L20 | TOPS: 284 -> 273 | Hurt |
| use SageAttn2-4b rather than full-precision attention | Llama3.1 (8B) · Llama3.1 | MMLU accuracy: 0.635 -> 0.607 | Hurt |
| use SageAttn2-8b rather than full-precision attention | Llama3.1 (8B) · Llama3.1 | MMLU accuracy: 0.635 -> 0.634 | No clear effect |
| use SageAttn2-4b rather than full-precision attention | CogvideoX (1.5-5B) · CogvideoX (1.5-5B) | VQA-a: 70.231 -> 57.729 | Hurt |
| use SageAttn2-8b rather than full-precision attention | CogvideoX (1.5-5B) · CogvideoX (1.5-5B) | VQA-a: 70.231 -> 69.492 | Hurt |
Surprising findings
- The FP32 accumulator for FP8 mma is actually FP22 (1 sign, 8 exponent, 13 mantissa bits), causing accuracy loss in PV products.
- Per-tensor INT4 quantization of Q and K makes CogVideoX produce a completely blurry video and Llama3 drops to random-guess MMLU accuracy (25%).
Limitations
- INT4 variant (SageAttn2-4b) shows minor metric losses on some models, while the 8-bit variant is needed for near-lossless results.
- No INT4 tensor core support on Hopper, so the INT4 kernel is for RTX4090/L20-class GPUs; Hopper uses the INT8 variant.
- Smoothing Q adds about 3.7% kernel overhead (283 to 273 TOPS on L20).
Experimental setup
Datasets: WikiText LAMBADA MMLU LongBench open-sora prompt sets MJHQ-30K ImageNet Librispeech
Benchmarks: WikiText LAMBADA MMLU LongBench open-sora prompts (VBench-style video metrics) MJHQ-30K ImageNet InfiniBench Needle-in-a-Haystack Librispeech
Metrics: perplexity accuracy CLIPSIM VQA-a VQA-t FID cosine similarity relative L1 RMSE TOPS
Baselines: SageAttention FlashAttention2 FlashAttention3 (fp8) xformers HadmdAttn SmoothAttn
Builds on: SageAttention FlashAttention SmoothQuant QServe QuaRot
Closest papers by meaning
- SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration (ICLR 2025 · similarity 0.92)
- SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training (NeurIPS 2025 · similarity 0.89)
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (NeurIPS 2024 · similarity 0.88)
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (ICLR 2024 · similarity 0.88)
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS 2025 · similarity 0.86)
- Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.