Research map / Large language models / Reasoning and chain of thought
SnapKV: LLM Knows What You are Looking for Before Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SnapKV, a fine-tuning-free method that compresses prompt KV caches per attention head by voting with an observation window at the end of the prompt, with pooling for clustering, yielding faster decoding and lower memory on long contexts.
Problem
KV cache growth with long prompts makes LLM decoding slow and memory-hungry; prior KV eviction methods mainly compress generated tokens and neglect prompt KV compression.
Approach
Attention patterns over prompt tokens identified by the last window of the prompt (observation window) remain consistent during generation, so per-head top-k voting on that window selects prefix KVs to keep, pooled for clustering, with the observation window KVs appended, yielding a fixed-size prompt cache with no fine-tuning.
Key results
- On LWM-Text-Chat-1M at 16K tokens with batch size 2, SnapKV decodes at under 40 ms/token versus over 100 ms for baseline (~3.6x speedup), and extends the OOM limit from 16K to 131K (~8.2x memory efficiency).
- SnapKV with 1024 prompt KV entries retains comparable LongBench scores across 16 datasets for four models, with an average compression of 92% at 1024 on ~13K-token inputs.
- On Command-R, SnapKV with a 4096 KV cache at 128K context shows -0.5% change on Needle-in-a-Haystack (9.866 vs 9.819).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compress Mistral prompt KV cache to 1024 tokens | Mistral-7B-Instruct-v0.2 · LongBench | score: 25.54 vs 26.82 on NarrativeQA; 40.94 vs 42.77 on HotpotQA; 86.48 vs 86.23 on TriviaQA | Mixed |
| Compress Mistral prompt KV cache to 4096 tokens | Mistral-7B-Instruct-v0.2 · LongBench Qasper | score: 33.06 -> 33.36 | Helped |
| Use SnapKV instead of H2O at 4096-token cache capacity | Mistral-7B-Instruct-v0.2 · LongBench NarrativeQA | score: 22.61 -> 25.54 | Helped |
| Apply SnapKV with a 4096-token KV cache | Command-R · Needle-in-a-Haystack | score: 9.866 -> 9.819 (-0.5%) | No clear effect |
| Apply SnapKV | Command-R · RAG Citation | F1 score: -1.2% | Hurt |
| Apply SnapKV | Command-R · RAG End-to-end | F1 score: -2.1% | Hurt |
| Apply SnapKV to generation with 200 sampled documents | Command-R · RAG generation, average over ground-truth positions | accuracy: +5.4% | Helped |
| Apply SnapKV to generation with 30 sampled documents | Command-R · RAG generation, average over ground-truth positions | accuracy: -1.7% | Hurt |
| Compress prompt KV cache | LWM-Text-Chat-1M · 16k input, batch size 2 | decoding speed: approximately a 3.6x speedup | Helped |
| Compress prompt KV cache | LWM-Text-Chat-1M · batch size 2 | maximum input length: approximately 8.2x improvement | Helped |
| Combine SnapKV with Medusa parallel decoding | Mistral-7B-Instruct-v0.2 · QASPER, 10k sequence length | generation speed: 1.3x speedup | Helped |
| Add pooling-based clustering to selected attention positions (ablation) | SnapKV on Mistral-7B-Instruct-v0.2 · LongEval-Lines | retrieval accuracy | Helped |
Surprising findings
- SnapKV sometimes outperforms the full-KV baseline (e.g., on some LongBench tasks and in the Command-R bioasq RAG test at 200 documents, +5.4% average), which the authors attribute to reduced noise.
- Instruction placement (before or after the context) does not noticeably change the hit rate of selected prefix features.
Limitations
- Only compresses generation-stage KV cache; does not reduce prompt-inference (prefill) cost.
- Cannot extend a model's inherent long-context capability if the base model performs poorly on long contexts.
- Results on Command-R RAG and internal benchmarks rely on proprietary/internal datasets that are not public.
Experimental setup
Datasets: UltraChat LongBench LongEval-Lines Needle-in-a-Haystack QMSum OpenReview SPACE BioASQ HotpotQA QASPER
Benchmarks: LongBench Needle-in-a-Haystack LongEval-Lines NarrativeQA BioASQ HotpotQA QASPER
Metrics: task score (LongBench) retrieval accuracy decoding latency (ms/token) memory footprint F1 score hit rate
Baselines: H2O All KV (full cache) StreamingLLM FastGen ScissorHands
Builds on: H2O StreamingLLM Medusa
Closest papers by meaning
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.92)
- CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.92)
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference (ICML 2025 · similarity 0.91)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.91)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.91)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.