Research map / Architectures and efficiency / Attention and transformers

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Converts a full-attention LLM into a head-wise sparse model by routing retrieval heads through a 16-dim pre-RoPE top-p selector trained in two stages, reaching near-lossless accuracy with up to 9.36x prefill speedup.

Problem

Long-context inference with full attention is bottlenecked by quadratic cost; native sparse training or heuristic token eviction trade off efficiency, training cost, and accuracy. The paper asks how little surgery can turn a full-attention model into a highly sparse one while preserving capability.

Approach

Identify retrieval heads offline via needle-based attention scores, keep full KV only for them, use sliding window with sinks for local heads, and select tokens for retrieval heads with a trained low-dimensional pre-RoPE projection and a query-adaptive top-p threshold, followed by logit self-distillation.

Key results

• On LongBench, the top-p variant averages 54.24% versus 53.80% for full attention and 53.30% for the top-k variant.
• On RULER 64K, top-p averages 85.49% versus 86.23% for full attention, while top-k drops to 70.53%.
• Attention layer prefill speedup over FlashAttention-2 grows from 2.83x at 32K to 9.36x at 1M; decode speedup reaches 2.01x at 1M.

Surprising findings

• Dimension 4 gave the highest end-to-end accuracy, but only because its weak fit caused the selector to retain more tokens, lowering actual sparsity.
• Top-p sometimes exceeds full attention on LongBench average (54.24% vs 53.80%).

Limitations

• Depends on stable retrieval/local head specialization; may degrade for models with weaker specialization or domain shift.
• Retrieval heads still use dense attention during prefill, so sparsification is incomplete.
• Experiments focus on the Qwen3 family and long-context/reasoning workloads.

Experimental setup

Datasets: FineWeb Dolma 3 Longmimo Mix

Benchmarks: LongBench RULER AIME24 AIME25 MMLU-PRO

Metrics: accuracy compute sparsity memory sparsity attention mass recall speedup

Baselines: RazorAttention MInference FlexPrefill Quest SnapKV FlashAttention-2 static top-k

Builds on: RoPE DuoAttention RazorAttention FlashAttention-2 SnapKV

Closest papers by meaning

• FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.92)
• Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models (NeurIPS 2026 · similarity 0.91)
• Efficient Low Rank Attention for Long-Context Inference in Large Language Models (NeurIPS 2025 · similarity 0.90)
• QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.90)
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.90)
• More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding (NeurIPS 2026 · similarity 0.90)