Research map / Architectures and efficiency / Attention and transformers
XAttention: Block Sparse Attention with Antidiagonal Scoring
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes XAttention, a training-free block-sparse attention method that scores attention blocks by strided antidiagonal sums plus threshold-based block selection and per-head threshold tuning, achieving near-full accuracy with up to 13.5x prefill attention speedup.
Problem
Quadratic attention cost in long-context transformers makes prefill expensive; existing block-sparse methods have costly block-importance estimation that trades off accuracy and efficiency.
Approach
Use the sum of strided antidiagonal values within each attention block as a cheap proxy for block importance, then select the minimal set of blocks whose softmax-normalized antidiagonal mass exceeds a threshold, with per-head thresholds tuned by dynamic programming.
Key results
- On RULER with Llama-3.1-8B-Instruct, XAttention (S=8) averages 88.47 vs 87.52 for full attention and 87.72 for FlexPrefill.
- Achieves up to 13.5x prefill attention speedup at 256K tokens (density 7.32%) relative to FlashInfer FlashAttention.
- On HunyuanVideo with tau=0.95 and 5-step warmup, reaches PSNR 23.5, SSIM 0.822, LPIPS 0.155 vs full attention at 45.5% density.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use antidiagonal scoring instead of random or diagonal scoring (ablation) | XAttention · RULER | accuracy: 82.53 -> 90.75; 76.47 -> 90.75 | Helped |
| Increase the antidiagonal stride to 64 (ablation) | XAttention · RULER Avg. | accuracy: 88.47 -> 81.21 | Hurt |
| Use stride 4 instead of stride 8 (ablation) | XAttention · RULER Avg. | accuracy: 88.47 -> 88.89 | Helped |
| Replace threshold block selection with Top K (ablation) | XAttention · RULER Avg. | accuracy: 88.47 -> 84.13 | Hurt |
| Replace threshold block selection with fixed Ratio (ablation) | XAttention · RULER Avg. | accuracy: 88.47 -> 85.42 | Hurt |
| Use dynamically predicted minimum thresholds instead of fixed τ=0.9 (ablation) | XAttention · RULER Avg. | accuracy: 84.96 -> 88.47 | Helped |
| Use stride 16 rather than stride 8 (ablation) | XAttention · RULER Avg. | accuracy: 88.47 -> 88.08 | Mixed |
| Raise the VBench threshold from 0.90 to 0.95 (ablation) | XAttention on HunyuanVideo · VBench | PSNR: 21.5 -> 23.5 | Helped |
| Raise the VBench threshold from 0.90 to 0.95 (ablation) | XAttention on HunyuanVideo · VBench | Density: 34.4 -> 45.5 | Hurt |
| Use five full-attention warmup steps before XAttention (ablation) | XAttention on HunyuanVideo · VBench | visual fidelity | Helped |
| Use XAttention instead of full attention | Llama-3.1-8B-Instruct · LongBench Avg. | score: 40.34 -> 40.60 | Helped |
| Use XAttention instead of full attention | QwenVL-2-7B · Video-MME Overall w/ | accuracy: 69.2 -> 69.1 | No clear effect |
Surprising findings
- XAttention outperforms full attention on some RULER lengths and on Video-MME long videos despite being sparse.
- Diagonal pattern at stride 16 drops 32k RULER score to 58.26 while antidiagonal stays at 90.64.
Limitations
- Video generation results lack comparison to other sparse baselines since they do not support non-causal attention.
- Video generation needed a 5-step full-attention warmup to avoid layout shifts.
- Minimum threshold prediction is optional and tuned per model via dynamic programming on benchmark accuracy.
Experimental setup
Datasets: RULER LongBench Video-MME VBench
Benchmarks: RULER LongBench Video-MME VBench
Metrics: accuracy density attention speedup PSNR SSIM LPIPS
Baselines: FlashAttention MInference FlexPrefill SeerAttention
Builds on: MInference FlexPrefill FlashAttention
Closest papers by meaning
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS 2025 · similarity 0.89)
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.88)
- SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration (ICLR 2025 · similarity 0.88)
- PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers (NeurIPS 2026 · similarity 0.88)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.87)
- Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.