Research map / Architectures and efficiency / Attention and transformers

SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference

ICML 2025 · Architectures and efficiency › Attention and transformers · approach: Block-sparse attention kernels · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SpargeAttn, a training-free, universal sparse and quantized attention that predicts sparse blocks via selective token compression and skips further products with a warp-level sparse online softmax, accelerating language, image and video models without end-to-end metric loss.

Problem

Attention's quadratic cost dominates inference latency for long sequences, but existing sparse attention methods are tied to specific models or patterns, or require training, and often degrade end-to-end quality or give little speedup; a universal, accurate, low-overhead training-free sparse attenti

Approach

Two-stage online filter on FlashAttention tiling: (1) predict block-sparse mask by compressing only high self-similarity Q/K blocks into mean tokens and applying a TopCdf selection, always computing non-self-similar "fix" blocks; (2) a warp-level sparse online softmax skips P~V products when the local row max is far below the running max; plus HilbertCurve permutation for visual tokens and integra

Key results

• On Llama3.1 at 128K, SpargeAttn reaches speed 708.1 vs 156.9 for full attention with WikiText ppl 6.020 vs 6.013 and NIAH 0.909 vs 0.907.
• On Mochi, SpargeAttn gives 1.83x end-to-end speedup (1897s to 1037s on L40) with comparable video quality metrics.
• On Flux, SpargeAttn achieves speed 280.3 vs 158.2 for full attention, with FID 163.982 vs 166.103 and ImageReward 0.9207 vs 0.8701.

Measured effects reported in the paper

Surprising findings

• SpargeAttn slightly improves long-context LLM performance over full attention (e.g., NIAH 0.909 vs 0.907 and LongBench 39.058 vs 38.682), which the authors attribute to sparse focus on relevant information.
• Sparsity increases with sequence length on Llama3.1 (6.8% at 8K to 54% at 128K) under a fixed accuracy bound.

Limitations

• Hyperparameters (tau, theta, lambda) must be searched per layer using calibration inputs and L1 thresholds.
• Sparsity and speedup depend on the model; e.g., video and image gains vary and sparsity varies across layers and heads.

Experimental setup

Datasets: WikiText LongBench InfiniteBench Needle-in-a-Haystack open-sora prompt sets COCO annotations

Benchmarks: WikiText LongBench InfiniteBench En.MC Needle-in-a-Haystack open-sora prompts (VQA, CLIPSIM, FScore) COCO (FID, CLIP, ImageReward)

Metrics: perplexity LongBench score retrieval accuracy CLIPSIM CLIP-T VQA-a VQA-t FScore FID CLIP score

Baselines: MInference FlexPrefill Full-Attention

Builds on: FlashAttention SageAttention online softmax MInference

Closest papers by meaning

• Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction (NeurIPS 2025 · similarity 0.89)
• AdaSplash: Adaptive Sparse Flash Attention (ICML 2025 · similarity 0.88)
• SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration (ICLR 2025 · similarity 0.88)
• Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.88)
• XAttention: Block Sparse Attention with Antidiagonal Scoring (ICML 2025 · similarity 0.87)
• Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.87)