Research map / Architectures and efficiency / Attention and transformers
RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RazorAttention, a training-free KV cache compression that keeps full cache for retrieval heads (identified via induction/echo scores) and compresses remote tokens of other heads into a single compensation token, achieving ~3x KV reduction.
Problem
KV cache growth is the main memory and compute bottleneck for long-context LLM inference, and importance-based token dropping irreversibly discards information that later queries may need.
Approach
Use head-wise caching: a small set of retrieval heads (found via induction and echo scores on repeated random tokens) keep the full KV cache, while all other heads keep only recent tokens and sink tokens, with the dropped tokens averaged into one compensation key/value; the head-wise criterion does not need attention maps, so it works with FlashAttention.
Key results
- On LongBench with Qwen1.5-7B-Chat, RazorAttention averages 35.87 vs 36.03 for full KV and 34.16 for H2O under roughly 3.125x compression.
- On Llama3-8B-Instruct, RazorAttention averages 34.86 vs 35.44 full KV and 32.89 H2O.
- Needle-in-a-haystack with Qwen1.5-7B-Chat: 14% induction + 1% echo heads scores 86.59% vs 87.05% baseline, while 5% induction heads drops to 69.54%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compress the KV cache with RazorAttention at the reported 3X compression ratio | Qwen1.5-7B-Chat · LongBench average | score: 36.03 -> 35.87 | No clear effect |
| compress the KV cache with RazorAttention at the reported 3X compression ratio | Qwen1.5-72B-Chat · LongBench average | score: 46.15 -> 45.97 | No clear effect |
| compress the KV cache with RazorAttention at the reported 3X compression ratio | Llama3-8B-Instruct · LongBench average | score: 35.44 -> 34.86 | No clear effect |
| compress the KV cache with RazorAttention at the reported 3X compression ratio | Baichuan2-13B · LongBench average | score: 36.41 -> 36.45 | No clear effect |
| protect retrieval heads rather than random heads while keeping only recent 4K tokens in the rest | Qwen1.5-7B-Chat · MultiFieldQA-en | score: 40.7% -> 45.48% | Helped |
| increase protected induction heads from 5% to 8% while protecting 1% echo heads (ablation) | RazorAttention · Needle in a Haystack | score: 69.54% -> 78.40% | Helped |
| increase protected induction heads from 8% to 11% while protecting 1% echo heads (ablation) | RazorAttention · Needle in a Haystack | score: 78.40% -> 84.55% | Helped |
| increase protected induction heads from 11% to 14% while protecting 1% echo heads (ablation) | RazorAttention · Needle in a Haystack | score: 84.55% -> 86.59% | Helped |
| add a compensation token for the dropped KV cache (ablation) | RazorAttention · long-context tasks | accuracy | Helped |
| add 1% echo heads to the protected retrieval-head group (ablation) | RazorAttention on Llama2-7B-80K · Needle in a Haystack | retrieval performance | Helped |
Surprising findings
- Protecting randomly chosen heads gives no gain over no protection on MultiFieldQA-en, while protecting retrieval heads recovers most performance.
- Only about 1% echo heads added to induction heads markedly improves retrieval on Llama2-7B-80K.
Limitations
- Optimal number of retrieval heads may differ across models and may need tuning per model.
- Compression ratio capped at about 70% (roughly 3x); the authors believe it can be improved.
- Mechanism behind why attention heads differ and how retrieval heads operate on long inputs is not fully explained.
Experimental setup
Datasets: LongBench Needle In A Haystack
Benchmarks: LongBench Needle In A Haystack
Metrics: LongBench score needle retrieval accuracy
Baselines: H2O StreamingLLM
Builds on: H2O StreamingLLM FlashAttention induction heads
Closest papers by meaning
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.93)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.93)
- Training Free Exponential Context Extension via Cascading KV Cache (ICLR 2025 · similarity 0.92)
- KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction (NeurIPS 2025 · similarity 0.92)
- EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction (NeurIPS 2026 · similarity 0.92)
- Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.