Research map / Architectures and efficiency / Attention and transformers

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adds a trainable Forecast projection to block-sparse attention (InfLLM-V2 style) that predicts the next layer's top-k blocks one layer ahead, enabling CPU-to-GPU KV prefetch overlap and a cheaper per-GQA-group indexer, trained by KL distillation on frozen sparse-pretrained 8B models.

Problem

Long-context LLM decoding with sparse attention still faces KV cache capacity growth with CPU offloading (PCIe transfer bottleneck) and O(T^2) sparse block-selection cost that can dominate attention time.

Approach

A fourth per-layer projection (Forecast) is trained to predict layer l+1's top-k block selection from layer l, decoupling selection from the attention query so KV prefetch can overlap with compute; the Forecast indexer uses one head per GQA group and no softmax, and is trained via KL against the original selector's block-attention distribution with only the Forecast projections trained.

Key results

• On RULER, SparDA improves over the Sparse baseline at every length on NOSA-8B, widening from +1.7 at 32K to +4.3 at 128K.
• At 128K decode, SparDA reaches up to 1.69x speedup over offloaded Sparse on MiniCPM4.1-8B and up to 5.28x throughput over non-offload Sparse at larger batch sizes.
• Average accuracy across four benchmark families improves by +0.3 (MiniCPM4.1-8B) and +2.3 (NOSA-8B) over Sparse, with NOSA-8B reasoning gaining +6.5.

Surprising findings

• Using a finer target compression window (2,1) for indexer training, rather than the inference-matched (32,16), gives notably better accuracy.
• InfiniGen, a training-free lookahead method, degrades accuracy substantially on both models.

Limitations

• Accuracy is bounded by the base sparse attention backbone; SparDA is an add-on, not a new sparse pattern.
• Evaluated only on two 8B sparse-pretrained models; not applied to DSA/CSA-based larger models like DeepSeek-V3.2 or DeepSeek-V4.
• Non-prefetch variant and layer-0 KV retention cause OOM slightly earlier than Sparse at long contexts.

Experimental setup

Datasets: ProLong-64K

Benchmarks: HELMET LongBench RULER MATH-500 AIME 2024 AIME 2025

Metrics: accuracy prefill throughput (tok/s) decode throughput (tok/s)

Baselines: Dense Sparse (InfLLM-V2) InfiniGen

Builds on: InfLLM-V2 DeepSeek Sparse Attention InfiniGen NOSA MiniCPM4

Closest papers by meaning

• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.90)
• RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.90)
• ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference (ICML 2025 · similarity 0.90)
• FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.90)
• QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.89)
• SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024 · similarity 0.89)