Research map / Architectures and efficiency / Attention and transformers

More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAGA, an asymmetric GQA variant with fewer key heads than value heads paired with Atop-N sparse decoding, plus a logit-matching distillation conversion from GQA models, achieving over 2x decode speedups at long context.

Problem

Autoregressive LLM decoding is bottlenecked by attention memory and compute; under sparse top-N attention the query-key step becomes the dominant cost, so existing symmetric GQA head counts are suboptimal for sparse decoding.

Approach

Under top-N sparsity the probability-value cost is negligible, so key heads can be reduced independently of value heads (SAGA), keeping value capacity; key heads are merged from a GQA teacher via closed-form logit-matching initialization (min-weight perfect matching) followed by distillation.

Key results

• SAGA with Atop-N achieves 2.2x end-to-end decode speedup on Qwen2.5-1.5B and 2.1x on Llama 3.2-1B at 128K, batch size 1, over full-attention GQA baseline.
• Distilled SAGA student retains 96.3% of teacher's aggregate RULER score across 8K-32K contexts on Qwen2.5-1.5B.
• From-scratch SAGA (1,8) on SmolLM2-360M reaches 42.7% validation accuracy vs 42.7%-range for (8,8) GQA while using a little over half the KV cache.

Surprising findings

• Atop-N on Llama3.1-70B-Instruct leaves GPQA Main accuracy unharmed at N=512 (0.4241 vs baseline 0.4241) while GPQA Diamond degrades (0.3586 vs 0.4343).
• Flash attention provides little benefit during autoregressive decode, so a standard compiled full-attention baseline was used.

Limitations

• Relies on simple Atop-N method rather than hardware-aware sparse kernels, limiting speedups.
• Quality experiments limited to models up to 1.5B parameters; no full-scale pretraining validation.
• Distilled SAGA students keep teacher architecture except reduced key heads, so capacity loss is not compensated; short-context speedups are negative due to overhead.

Experimental setup

Datasets: SlimPajama HotpotQA SQuAD LongAlign-10k LongAlpaca-12k RULER HellaSwag WikiText GPQA

Benchmarks: RULER HellaSwag WikiText GPQA Main GPQA Diamond

Metrics: accuracy decode latency speedup perplexity KL divergence cross-entropy

Baselines: GQA full attention SW+S H2O KIVI Sigma exact top-N

Builds on: Grouped-Query Attention top-N attention RULER

Closest papers by meaning

• Loki: Low-rank Keys for Efficient Sparse Attention (NeurIPS 2024 · similarity 0.91)
• TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention (ICLR 2025 · similarity 0.90)
• Efficient Low Rank Attention for Long-Context Inference in Large Language Models (NeurIPS 2025 · similarity 0.90)
• When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models (ICML 2024 · similarity 0.90)
• Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (NeurIPS 2024 · similarity 0.90)
• Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.90)