Research map / Architectures and efficiency / Attention and transformers
HashAttention: Semantic Sparsity for Faster Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HashAttention, which learns mappings that hash keys/values and queries into 32-bit Hamming-space signatures to identify pivotal tokens for sparse SDPA, reaching 16x-32x sparsity with minimal quality loss.
Problem
Scaling attention to long contexts is costly because SDPA must read the entire KV cache per decoded token; existing top-k sparse attention methods either lose quality under heuristic token selection or need substantial auxiliary memory and are not GPU-friendly.
Approach
Frames pivotal-token identification as maximum inner product search reduced to cosine similarity, then learns independent key-value and query encoders that map vectors to compact bit signatures so top tokens are found via XOR+popcount Hamming distance on GPU.
Key results
- At 16x sparsity, HashAttention on LongBench averages 48.00 vs 48.78 for full Llama-3.1-8B-Instruct, with 32 bits per token per head of auxiliary memory.
- On A100 at 32x sparsity, attention latency is reduced up to 4.3x in GPT-FAST and 2.54x in FlashDecode, and end-to-end throughput rises up to 3.12x for GPT-FAST.
- Under a 32-bit budget on RULER@16K, HashAttention at 16x reaches 89.53 average vs 90.66 full, while LSH at 32 bits drops to about 48.9 on LongBench subsets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HashAttention instead of Quest | Llama-3.1-8B-Instruct sparse attention · LongBench average | score: 63.33 -> 64.15 | Helped |
| Apply 16x HashAttention sparsity | Llama-3.1-8B-Instruct · LongBench average | score: 48.78 -> 48.00 | Hurt |
| Apply 16x HashAttention sparsity | Llama-3.1-8B-Instruct · RULER@16K average | score: 90.66 -> 89.53 | Hurt |
| Use 32x HashAttention sparsity | GPT-FAST attention · inference | latency: up to 4.3× | Helped |
| Use 32x HashAttention sparsity | FlashDecode attention · inference | latency: 2.54× | Helped |
| Use 32x HashAttention sparsity | GPT-FAST with Llama-3.1-8B · inference | throughput: 3.12× | Helped |
| Replace learned signatures with 32-bit LSH signatures (ablation) | HashAttention · LongBench average at 16x compression | score: 75.0675 -> 48.8975 | Hurt |
| Increase LSH signature width from 32 bits to 256 bits (ablation) | LSH sparse attention · LongBench average at 16x compression | score: 48.8975 -> 69.7875 | Helped |
| Increase HashAttention signature width from 16 to 32 bits (ablation) | HashAttention top-64 prediction · 250 × 32K samples | cross-entropy loss: 0.243 -> 0.212 | Helped |
| Increase HashAttention signature width from 32 to 64 bits (ablation) | HashAttention top-64 prediction · 250 × 32K samples | cross-entropy loss: 0.212 -> 0.204 | Helped |
Surprising findings
- Certain sparsity levels slightly improve quality over the full model on some tasks, which the authors attribute to avoiding distractor tokens.
- Random-projection LSH needs more than 1000 bits to approach learned 32-bit HashAttention signatures on top-k prediction.
Limitations
- Each model requires its own trained hash mappings, unlike training-free methods.
- Evaluation restricted to KV cache resident on GPU; CPU-offloaded long-context settings not evaluated.
- Quality drops more on Chinese LongBench tasks when trained only on English data.
Experimental setup
Datasets: OpenWebText LongBench RULER
Benchmarks: LongBench RULER
Metrics: task score recall of top-k pivotal tokens attention latency end-to-end throughput cosine similarity of query-key embeddings
Baselines: StreamingLLM H2O InfLLM Double Sparsity Quest Oracle top-k LSH (random signed projections)
Builds on: Double Sparsity Quest vLLM PagedAttention StreamingLLM
Closest papers by meaning
- Rethinking Semantic ID Construction for Generative Recommendation: SimHash with Parallel Decoding and Semantic Alignment (NeurIPS 2026 · similarity 0.85)
- RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (NeurIPS 2025 · similarity 0.85)
- HyperAttention: Long-context Attention in Near-Linear Time (ICLR 2024 · similarity 0.85)
- OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference (ICML 2025 · similarity 0.85)
- Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity (NeurIPS 2025 · similarity 0.85)
- Efficient Low Rank Attention for Long-Context Inference in Large Language Models (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.