Research map / Architectures and efficiency / Quantization
ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZipCache, a mixed-precision KV cache quantization framework that uses channel-separable tokenwise quantization and normalized attention scores to identify salient tokens, with a probe-token approximation compatible with FlashAttention.
Problem
KV cache memory grows with batch size and context length, and prior adaptive KV cache compression methods that rank tokens by accumulated attention scores are biased toward early tokens and require full attention matrices, hurting accuracy at high compression and slowing inference.
Approach
Replace accumulated attention scores with column-normalized attention scores (divided by the count of nonzero entries per column) to remove the lower-triangular bias; quantize all tokens at adaptive bit-widths by saliency; approximate saliency from a small set of probe tokens so most tokens can use FlashAttention; and use channel-separable tokenwise quantization to cut quantization-parameter overh
Key results
- On GSM8k with Mistral-7B, ZipCache reaches 4.98x compression with 41.24% accuracy vs 41.62% FP16 (about 0.38% drop).
- On LLaMA3-8B at input length 4096, ZipCache reduces prefill latency by 37.3%, decoding latency by 56.9%, and GPU memory by 19.8% relative to MiKV.
- On the 200-line retrieval task with Mistral-7B, ZipCache improves accuracy over MiKV by 42%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use normalized attention scores instead of accumulated attention scores for token saliency | ZipCache · GSM8K CoT | accuracy (%): +18.27% | Helped |
| Use normalized attention scores instead of accumulated attention scores for token saliency | ZipCache · 200-line retrieval | accuracy (%): +10.0% | Helped |
| Approximate saliency from a subset of probes and use fast attention for other tokens | ZipCache · LLaMA3-8B, input length 4096 | prefill-phase latency: 37.3% reduction | Helped |
| Approximate saliency from a subset of probes and use fast attention for other tokens | ZipCache · LLaMA3-8B, input length 4096 | decoding-phase latency: 56.9% reduction | Helped |
| Approximate saliency from a subset of probes and use fast attention for other tokens | ZipCache · LLaMA3-8B, input length 4096 | GPU memory usage: 19.8% reduction | Helped |
| Add channel-separable tokenwise quantization for value cache (ablation) | KV cache quantization baseline · GSM8K | accuracy (%): 52.77 -> 54.74 | Helped |
| Replace groupwise quantization with channel-separable quantization (ablation) | KV cache quantization baseline · GSM8K | accuracy (%): 54.51 -> 54.74 | Helped |
| Use tokenwise quantization for both key and value instead of channel-separable value quantization (ablation) | KV cache quantization baseline · GSM8K | accuracy (%): 54.74 -> 49.81 | Hurt |
| Replace channel-separable tokenwise value quantization with tokenwise value quantization (ablation) | KV cache quantization baseline · GSM8K | accuracy (%): 54.74 -> 52.77 | Hurt |
| Use random+recent tokens as probes rather than random-only probes (ablation) | ZipCache · GSM8K | accuracy (%): 47.46 -> 52.08 | Helped |
| Use random+recent tokens as probes rather than special-token probes (ablation) | ZipCache · GSM8K | accuracy (%): 46.78 -> 52.08 | Helped |
| Use random+recent tokens as probes rather than recent-only probes (ablation) | ZipCache · GSM8K | accuracy (%): 51.10 -> 52.08 | Helped |
Surprising findings
- Accumulated attention scores systematically rank the final question tokens (which are salient for CoT tasks) as low saliency, which explains MiKV's accuracy collapse.
- Recent-token-only or special-token probe strategies underperform a random+recent hybrid probe selection.
Limitations
- Saliency ratio is manually specified before evaluation and cannot adapt automatically to the task dataset.
- Potential for misuse to generate malicious content, as with other generative models.
Experimental setup
Datasets: GSM8k HumanEval Line Retrieval
Benchmarks: GSM8k HumanEval Line Retrieval
Metrics: accuracy compression ratio prefill latency decoding latency GPU memory
Baselines: H2O GEAR KIVI MiKV FP16
Builds on: H2O KIVI GEAR MiKV FlashAttention
Closest papers by meaning
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.92)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.92)
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.91)
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (ICML 2024 · similarity 0.91)
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.91)
- AttentionPredictor: Temporal Patterns Matter for KV Cache Compression (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.