Research map / Large language models / Reasoning and chain of thought
KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KVzip, a query-agnostic KV cache eviction method that scores KV pairs by max cross-attention received while the LLM reconstructs its own context, enabling a compressed cache to be reused across diverse queries.
Problem
Long-context LLM KV caches grow with context length, causing memory and attention latency overhead; existing query-aware eviction methods overfit to the first query and degrade badly when the compressed cache is reused across multiple queries.
Approach
Score each context KV pair by the maximum attention it receives while the model is teacher-forced to repeat the context (context reconstruction), computed chunk-wise to remain linear in context length, then evict lowest-scoring pairs once per context independent of any query.
Key results
- KVzip reduces KV cache size by 3-4x and FlashAttention decoding latency by about 2x with negligible loss across QA, retrieval, reasoning, and code tasks with contexts up to 170K tokens.
- On SQuAD with LLaMA3.1-8B at 30% cache ratio, KVzip reaches 94.37% accuracy with Qwen2.5-7B (ablation), while SnapKV reaches 32.15% under reuse.
- KVzip consistently maintains accuracy when evicting up to 70% of the KV cache, whereas baselines degrade at around 10% eviction in multi-query settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compress the KV cache with KVzip | KVzip · Diverse queries and tasks | KV cache size: 3−4× reduction | Helped |
| compress the KV cache with KVzip | FlashAttention decoding · Diverse queries and tasks | latency: approximately 2× reduction | Helped |
| use KVzip rather than existing KV eviction methods in multi-query settings | KV eviction methods · Multi-query benchmarks | performance under cache compression | Helped |
| vary scoring chunk size (ablation) | KVzip · SCBench; LLaMA3.1-8B; 0.3 KV cache ratio | average performance difference: below 2% | No clear effect |
| replace the original repeat prompt with a paraphrased prompt (ablation) | KVzip · SQuAD; 30% KV cache ratio; Qwen2.5-7B | accuracy (%): 94.37 -> 94.45 | No clear effect |
| remove the repeat prompt (ablation) | KVzip · SQuAD; 30% KV cache ratio; Qwen2.5-7B | accuracy (%): 94.37 -> 94.25 | No clear effect |
| use full context reconstruction rather than only the first 10%, last 10%, or prompt alone for scoring (ablation) | KVzip · KV eviction performance across tasks | performance | Helped |
| omit Softmax normalization in importance scoring (ablation) | KVzip · Retr.KV; SCBench; LLaMA3.1-8B | compression ratio: approximately a 10% degradation | Hurt |
| use uniform rather than non-uniform KV head budgets (ablation) | KVzip · SQuAD; LLaMA3.1-8B | compression performance | Hurt |
| use uniform KV head-budget allocation (ablation) | KVzip · SQuAD; LLaMA3.1-8B | performance versus baseline | Helped |
Surprising findings
- After KV eviction, LLaMA3.1-8B answers a privacy-related query (phone number) that it refuses with the full cache.
- Reconstruction-based head-level scores from a single 88K-token book sample, obtained in about one minute, outperform DuoAttention's multi-hour optimized head scores.
Limitations
- No theoretical guarantees on compression-induced information loss.
- Compression incurs per-context scoring overhead (about 2x prefill), amortized only over multiple queries.
- Context-independent head-level eviction achieves lower compression than context-dependent eviction.
Experimental setup
Datasets: SQuAD GSM8K NIAH SCBench RULER
Benchmarks: SQuAD GSM8K NIAH SCBench RULER DecodingTrust
Metrics: accuracy relative performance to full cache attention latency KV cache size
Baselines: H2O SnapKV PyramidKV DuoAttention
Builds on: SnapKV PyramidKV H2O DuoAttention FlashAttention
Closest papers by meaning
- Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction (NeurIPS 2026 · similarity 0.93)
- KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse (NeurIPS 2025 · similarity 0.93)
- PatchKV: Weight Space Compensation of KV Cache (NeurIPS 2026 · similarity 0.92)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.92)
- Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.91)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.