Research map / Architectures and efficiency / Attention and transformers
Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Delta Attention, a training-free correction that adds the difference between dense attention on every γ-th query and sparse attention to sparse prefill outputs, recovering much of quadratic attention accuracy on long-context RULER.
Problem
Sparse attention prefill methods cut the quadratic cost of attention but cause a distributional shift in attention outputs, so decoding-time queries fail to align with the right prefill keys, degrading long-context accuracy.
Approach
Approximate the missing delta-shaped attention region as the difference between dense and sparse attention outputs, computed densely only for every γ-th query and reused for the following γ-1 rows, so the output is corrected without full quadratic cost.
Key results
- On RULER 131K with Llama 3.1 8B, Streaming LLM + Delta improves average accuracy from 44.25 to 83.06 (about 36 points), recovering 88% of Flash Attention 2 accuracy.
- Delta correction on top of Streaming LLM gives a more than 32x speedup over Flash Attention 2 for a single attention layer at 1M tokens, with gamma=64 (about 98.5% sparsity).
- On InfiniteBench with Llama 4 Scout, Streaming LLM + Delta raises average from 28.47 to 57.35 versus 69.11 for Flash Attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Δ correction to Streaming LLM | Llama 3.1 8B Instruct · RULER 131K | accuracy: 27.45 -> 64.40 | Helped |
| Add Δ correction to HiP | Llama 3.1 8B Instruct · RULER average | accuracy: 87 .77 -> 88 .76 | Helped |
| Add Δ correction to Streaming LLM | Llama 3.1 8B Instruct · PG19 Long QA | Long PPL: 7.02 (+1.91) -> 5.96 (+0.85) | Helped |
| Add Δ correction to HiP Attention | Llama 3.1 8B Instruct · PG19 Long QA | PPL: 3.48 (+0.15) -> 3.37 (+0.04) | Helped |
| Add Δ correction to Streaming LLM | Llama 3.1 8B Instruct · InfiniteBench 126K | average: +29%pt | Helped |
| Add Δ correction to HiP | Llama 4 Scout 109B · InfiniteBench 384K | average: +0.5%pt | Helped |
| Replace Δ correction with Recompute, which recomputes selected dense rows without propagating their difference to neighboring tokens (ablation) | Streaming LLM · RULER 131K | accuracy: 64.40 -> 52.67 | Hurt |
| Use linear interpolation to smooth Δ corrections (ablation) | Streaming LLM + Δ · RULER average | accuracy: 83.06 -> 83.22 | No clear effect |
| Use EMA smoothing with β = 0.5 (ablation) | Streaming LLM + Δ · RULER average | accuracy: 83.06 -> 82.85 | Hurt |
| Use EMA smoothing with β = 0.75 (ablation) | Streaming LLM + Δ · RULER average | accuracy: 83.06 -> 82.69 | Hurt |
| Use EMA smoothing with β = 0.95 (ablation) | Streaming LLM + Δ · RULER average | accuracy: 83.06 -> 82.86 | Hurt |
| Use the α,β,γ filter with α=0.05, β=1.25 × 10⁻⁴, γ=2.08 × 10⁻⁵ (ablation) | Streaming LLM + Δ · RULER average | accuracy: 83.06 -> 81.50 | Hurt |
Surprising findings
- Dense decoding over a sparse prefill fails badly on MultiKey-3 (0% vs 62% for quadratic attention), indicating the distributional shift breaks query-key alignment even with full decode attention.
- Recomputing only the dense rows without propagating the correction gives most of the gain on average but is weaker on long contexts, and MInference's public kernel is slower than Flash Attention in the authors' latency tests.
Limitations
- The approximation that delta terms are stable across a gamma-window is only validated empirically, not theoretically.
- A fixed hyperparameter gamma sets the query gap; no adaptive selection criterion is proposed.
- Latency results exclude MInference due to its public implementation not fully utilizing the GPU, and the benchmarks are limited to specific GPUs and models.
Experimental setup
Datasets: RULER PG19 long QA InfiniteBench RepoQA
Benchmarks: RULER PG19 Long QA (PPL and LongPPL) InfiniteBench RepoQA
Metrics: accuracy perplexity LongPPL latency cosine similarity Spearman rank correlation
Baselines: StreamingLLM HiP Attention MInference Flash Attention 2 Star Attention CacheBlend APE Recompute
Builds on: StreamingLLM HiP Attention MInference Star Attention FlashAttention-2
Closest papers by meaning
- SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference (ICML 2025 · similarity 0.89)
- SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024 · similarity 0.88)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.88)
- FlashBias: Fast Computation of Attention with Bias (NeurIPS 2025 · similarity 0.88)
- SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration (ICLR 2025 · similarity 0.88)
- Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.