Research map / Architectures and efficiency / Attention and transformers
TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TidalDecode, which runs full attention and exact top-k token selection in only two layers and reuses the selected tokens across all other layers (position persistent sparse attention), with periodic KV cache correction, to speed up long-context LLM decoding.
Problem
Long-context LLM decoding is memory-bound due to KV cache access, and existing selection-based sparse attention methods pick tokens independently per layer, which is costly and often misses the truly top-attended tokens.
Approach
Tokens with highest attention scores overlap strongly across consecutive layers, so only a few token selection layers (early and one middle layer) perform full attention with exact top-k selection, and all other layers reuse the pre-selected token set for sparse attention, plus a periodic KV cache correction step.
Key results
- On 100K-context Needle-in-the-Haystack with LLaMA-3-8B, TD+L13 reaches 92% accuracy at budget K=64 and 100% at K=128, versus 50% and 65% for Quest.
- On LongBench with Llama-3-8B-Instruct-Gradient-1048k at budget 4096, TD+L13 averages 32.86 versus 32.33 for full attention and 31.13 for Quest.
- TidalDecode reduces end-to-end decoding latency by up to 2.1x versus full attention and 1.2x versus Quest, per the abstract and efficiency results on Llama-2-7B on A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace per-layer token estimation with position-persistent token reuse | TidalDecode versus Quest · LongBench, LLaMA-3-8B-Instruct-Gradient-1048k, K=4096 | average score: 31.13 -> 32.86 | Helped |
| replace Quest with TidalDecode token selection | LLaMA-3.1-8B-Instruct · LongBench average, K=4096 | score: 37.76 -> 38.32 | Helped |
| replace full attention with TidalDecode | LLaMA-3.1-8B-Instruct · LongBench average, K=4096 | score: 39.02 -> 38.32 | Hurt |
| use sparse attention with TidalDecode instead of full attention | LLaMA-2-7B · decoding latency, 100K context, K=512 | latency: 45.70 -> 21.13 | Helped |
| use TidalDecode instead of Quest | LLaMA-2-7B · decoding latency, 100K context, K=512 | latency: 24.84 -> 21.13 | Helped |
| use layer 13 rather than layer 15 for token re-selection (ablation) | TidalDecode on LLaMA-3-8B · Needle-in-the-Haystack, 100K, K=128 | accuracy: 92% -> 100% | Helped |
| use layer 13 rather than layer 14 for token re-selection (ablation) | TidalDecode on LLaMA-3.1-8B · Needle-in-the-Haystack, 32K, K=64 | accuracy: 98% -> 100% | Helped |
| use layer 14 rather than layer 31 for token re-selection (ablation) | TidalDecode on LLaMA-3-70B · Needle-in-the-Haystack, 32K, K=64 | accuracy: 98% -> 82% | Mixed |
| choose the re-selection layer based on model and task (ablation) | TidalDecode · Needle-in-the-Haystack sensitivity studies | accuracy | Mixed |
| use layer 12 rather than layer 13 for token re-selection (ablation) | TidalDecode on LLaMA-3.2-3B-Instruct · Needle-in-the-Haystack, 10K, K=128 | accuracy: 94% -> 96% | Helped |
| use layer 7 rather than layer 13 for token re-selection (ablation) | TidalDecode on LongChat-7b-v1.5-32k · Needle-in-the-Haystack, 3K, K=32 | accuracy: 5% -> 80% | Helped |
Surprising findings
- TidalDecode matches or exceeds full attention on several LongBench tasks at sparsity levels, which the authors hypothesize is because token selection filters irrelevant information.
- Using a single token set for all layers without re-selection drops needle-in-the-haystack recall to under 20%, while a single middle re-selection layer raises it to nearly 40% on the analyzed setting.
Limitations
- KV cache correction was described but not used in the reported evaluations, so its benefit for long generation is untested.
- The optimal token re-selection layer varies by model and must be chosen via sensitivity sweeps.
Experimental setup
Datasets: PG-19 PG-19-mini
Benchmarks: Needle-in-the-Haystack PG-19 LongBench
Metrics: accuracy perplexity F1 end-to-end decoding latency
Baselines: Quest H2O TOVA StreamingLLM FlashInfer full attention
Builds on: Quest FlashInfer
Closest papers by meaning
- More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding (NeurIPS 2026 · similarity 0.90)
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.89)
- AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism (ICML 2025 · similarity 0.89)
- When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models (ICML 2024 · similarity 0.88)
- Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions (ICLR 2025 · similarity 0.88)
- Optimized Multi-Token Joint Decoding With Auxiliary Model for LLM Inference (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.