Research map / Architectures and efficiency / Attention and transformers
Value-Guided KV Compression for LLMs via Approximated CUR Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CurDKV, a value-centric KV cache compression method that selects keys and values via CUR-style leverage scores (approximated with Gaussian projections), plus an adaptive variant AdaCurDKV, evaluated on LongBench and Ruler with LLaMA-3.1-8B and Mistral-7B.
Problem
KV cache compression methods rank tokens by query-key attention scores, which ignores the contribution of value vectors to the attention output softmax(QK^T)V, leading to loss in output fidelity under aggressive compression.
Approach
Select retained keys and values using leverage scores from CUR/SVD of the original key and value matrices (product of key and value leverage scores, approximated via random Gaussian projections), rather than attention scores; also an adaptive head-wise budget variant (AdaCurDKV).
Key results
- On LLaMA-8B at 90% compression on LongBench, CurDKV averages 35.7 vs SnapKV 33.7 (+2.0 points); at 30% it reaches 48.9 vs 45.3.
- On Ruler NIAH at 30% compression, CurDKV averages 98.7% vs SnapKV 80.4% and ChunkKV 87.0% for LLaMA-8B.
- AdaCurDKV averages 39.1% on Ruler NIAH at 90% compression for LLaMA-8B vs 24.4% for AdaSnapKV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CUR-based combined key-value leverage scores instead of attention-based token ranking | KV cache compression · LongBench 30% compression, LLaMA-3.1-8B-Instruct | average score: 45.3 -> 48.9 | Helped |
| use CUR-based combined key-value leverage scores instead of attention-based token ranking | KV cache compression · LongBench 30% compression, Mistral-7B-Instruct | average score: 42.1 -> 45.6 | Helped |
| use adaptive head-wise budget allocation | CurDKV · Ruler 90% compression, LLaMA-3.1-8B-Instruct | average accuracy: 34.7 -> 39.1 | Helped |
| use adaptive head-wise budget allocation | CurDKV · Ruler 90% compression, Mistral-7B-Instruct | average accuracy: 7.8 -> 3.9 | Hurt |
| use value-only or combined key-value leverage scores instead of key-only scores (ablation) | CurDKV · LongBench 90% compression | median accuracy | Helped |
| add Gaussian random projections for approximate leverage scores (ablation) | CurDKV · LongBench 90% compression | median accuracy | Helped |
Surprising findings
- Attention-score-based eviction loss is not always correlated with attention score, per the authors' analysis.
- Mistral shows marginal underperformance on some NIAH subtasks at 90% compression, attributed to sliding-window attention reducing head specialization.
Limitations
- Static token selection during prefill only; cannot adapt to information needed during generation.
- Marginal underperformance on some Mistral NIAH subtasks at 90% compression.
- Baselines were all run in-house, and code is only promised upon acceptance.
Experimental setup
Datasets: LongBench Ruler
Benchmarks: LongBench Ruler
Metrics: accuracy average score fidelity Friedman statistic generation latency prefill latency memory usage
Baselines: SnapKV ChunkKV StreamingLLM Knorm AdaSnapKV
Builds on: SnapKV AdaKV H2O StreamingLLM
Closest papers by meaning
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.91)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.91)
- EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction (NeurIPS 2026 · similarity 0.91)
- SALS: Sparse Attention in Latent Space for KV Cache Compression (NeurIPS 2025 · similarity 0.91)
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.91)
- KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.