Research map / Large language models / Instruction and fine-tuning
SparQ Attention: Bandwidth-Efficient LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SparQ Attention, a training-free inference method that reduces KV-cache memory transfers by selecting top query components for approximate scores, fetching only top-k keys/values, and adding a mean-value correction.
Problem
LLM token generation with long, batched inputs is bottlenecked by memory bandwidth from reading the full KV cache at each decoding step.
Approach
Approximate attention scores using only the r largest-magnitude query components to read a slice of the keys, fetch full keys/values only for the top-k positions, and reallocate the unattended softmax mass to a running mean value vector, all applied to off-the-shelf pretrained models without fine-tuning.
Key results
- On Llama 2 13B at 1/8 compression, SparQ retains SQuAD accuracy of 74.9% vs 80.8% dense, while H2O reaches 63.0% and LM-Infinite 30.1%.
- On Llama 3 8B at 1/8 compression, SparQ keeps SQuAD at 78.3% (dense 81.2%) and CNN/DailyMail ROUGE-L at 23.4 (dense 23.4).
- On a single H100 with Llama 2 7B at 1/8 compression, end-to-end speedups are reported; on A100 microbenchmarks the SparQ Triton kernel reaches 3.02x over dense at S=4096, batch 64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SparQ instead of H₂O at 1/8 compression | Llama 2 13B · SQuAD | accuracy: 63.0 -> 74.9 | Helped |
| Use SparQ instead of H₂O at 1/8 compression | Llama 2 13B · TriviaQA | accuracy: 78.4 -> 78.2 | Hurt |
| Use SparQ instead of H₂O at 1/8 compression | Llama 2 13B · CNN/DailyMail | ROUGE-L: 20.3 -> 21.6 | Helped |
| Use SparQ instead of H₂O at 1/8 compression | Llama 2 13B · WikiText-103 | BPC: 0.64 -> 0.70 | Hurt |
| Use SparQ instead of LM-Infinite at 1/8 compression | Llama 2 13B · Repetition | characters before divergence: 29 -> 190 | Helped |
| Replace LM-Infinite or H₂O with SparQ at 1/4 compression | LLaMA-2-7B-32K · Needle in a haystack, 8k–16k | accuracy: 23.5% -> 100%; 5.9% -> 100% | Helped |
| Replace LM-Infinite or H₂O with SparQ at 1/8 compression | LLaMA-2-7B-32K · Needle in a haystack, 8k–16k | accuracy: 11.8% -> 79.4%; 2.9% -> 79.4% | Helped |
| Set approximate softmax temperature to the proposed query-dependent temperature (ablation) | SparQ Attention · Llama 2 7B ablation tasks | task performance | Helped |
| Do not reallocate attention mass to the mean value (α=0) (ablation) | SparQ Attention · Llama 2 7B ablation tasks | task performance | Hurt |
| Reduce the number of selected key-value positions k (ablation) | SparQ Attention · Llama 2 7B ablation tasks | task performance | Hurt |
| Reduce the query rank r (ablation) | SparQ Attention · Llama 2 7B ablation tasks | task performance | Hurt |
| Use the top-r query components rather than the first r components (ablation) | SparQ Attention · Llama 2 7B attention-score approximation | top-k agreement | Helped |
Surprising findings
- Some heads' query vectors are heavy-tailed, so keeping only the top-r query magnitudes gives high top-k overlap with exact scores even at r=d_h/8.
- Removing the attention-reallocation step (Step 3) hurt performance for GQA models, so it was omitted for Llama 3 and Mistral.
Limitations
- Storing K twice increases KV cache memory by 50%.
- Speedups are limited for small batch sizes and short sequences on GPU due to extra kernel complexity.
- Evaluation focuses on tasks of 1k to 2k tokens, with limited long-context evaluation beyond SQuAD scaling and needle tests.
Experimental setup
Datasets: SQuAD v1.1 TriviaQA CNN/DailyMail WikiText-103 Tiny-Shakespeare Paul Graham essays
Benchmarks: SQuAD 1-shot TriviaQA 0-shot CNN/DailyMail WikiText-103 Text Repetition Needle in a haystack
Metrics: exact match accuracy ROUGE-L bits per character repetition length tokens per second speedup top-k agreement
Baselines: H2O LM-Infinite FlexGen oracle top-k random low-rank projection
Builds on: H2O FlexGen LM-Infinite grouped-query attention
Closest papers by meaning
- Star Attention: Efficient LLM Inference over Long Sequences (ICML 2025 · similarity 0.93)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.92)
- Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity (NeurIPS 2025 · similarity 0.91)
- FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.91)
- LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning (ICML 2024 · similarity 0.90)
- CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.