Research map / Architectures and efficiency / RNNs and normalization
CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSR, which compresses the LLM KV cache into sparse indices and coefficients over a learned dictionary (NeuralDict), reaching roughly 1-2 bit equivalent storage with accuracy comparable to 2-4 bit KIVI and GEAR on LongBench.
Problem
The KV cache grows linearly with context length and dominates memory during long-context LLM inference; existing KV quantization struggles below 2 bits per channel.
Approach
Treat KV vectors as signals to be sparsely coded over a dictionary: an offline dictionary (shared across adjacent layers with similar JS-divergence-based grouping) is learned by a single linear layer trained with MSE plus a diversity regularizer and sparse matching pursuit, and each token's KV is stored as s indices and coefficients.
Key results
- On Llama2-7B-chat, CSR-16 (about 4-bit equivalent) averages 37.58 on LongBench versus 37.26 FP16 and 37.19 for KIVI-4.
- CSR-8 (about 2-bit equivalent) averages 36.60 versus KIVI-2 at 37.17 on Llama2-7B-chat, so the two are comparable.
- CSR-4 (about 1-bit equivalent) averages 33.29 on Llama2-7B-chat, roughly 8% below FP16 per the authors' statement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CSR-16 instead of FP16 KV cache | Llama2-7B-Chat · LongBench Avg | score: 37.26 -> 37.58 | Helped |
| Use CSR-16 instead of 4-bit KIVI or GEAR | Llama2-7B-Chat · LongBench Avg | score: GEAR 37.42; KIVI-4 37.19; CSR-16 37.58 | Helped |
| Use CSR-8 instead of KIVI-2 | Llama2-7B-Chat · LongBench Avg | score: KIVI-2 37.17; CSR-8 36.60 | Mixed |
| Reduce KV cache to CSR-4 (1 bit) | Llama2-7B-Chat · LongBench Avg | score: 37.26 -> 33.29 | Hurt |
| Increase Value cache split count s_n from 1 to 4 (ablation) | NeuralDict · Value Cache, size 8192, s=4 | MSE loss: 0.259 -> 0.093 | Helped |
| Increase Key cache split count s_n from 1 to 4 (ablation) | NeuralDict · Key Cache, size 8192, s=4 | MSE loss: 0.079 -> 0.052 | Helped |
| Increase offline dictionary size from 1024 to 2048 (ablation) | NeuralDict · Llama3-8B-Instruct Key Cache | MSE loss: 0.1023 -> 0.0662 | Helped |
| Increase offline dictionary size from 1024 to 2048 (ablation) | NeuralDict · Llama3-8B-Instruct Value Cache | MSE loss: 0.1935 -> 0.1358 | Helped |
| Add diversity regularization to NeuralDict (ablation) | Llama2-7B NeuralDict · Converged training | MSE loss: 0.0572 -> 0.0561 | Helped |
| Add diversity regularization to NeuralDict (ablation) | Llama3-8B NeuralDict · Converged training | MSE loss: 0.0702 -> 0.0662 | Helped |
| Add online dictionary, size 8192 (ablation) | CSR-16 on Llama2-7B-Chat · LongBench Avg | score: 37.12 -> 37.58 | Helped |
| Increase online dictionary size from 8192 to 16384 (ablation) | CSR-16 on Llama2-7B-Chat · LongBench Avg | score: 37.58 -> 37.61 | No clear effect |
Surprising findings
- Removing the online dictionary (online size 0) drops CSR-8 on Llama2-7B-chat from 36.60 to 35.21, while for CSR-16 the effect is much smaller.
Limitations
- Offline dictionary training on a calibration corpus is time-consuming.
- Performance drops noticeably at 1-bit equivalent (CSR-4), especially on QA tasks like multifieldqa and Qasper.
- Evaluated only on LongBench and three 7-8B models.
Experimental setup
Datasets: wikitext LongBench
Benchmarks: LongBench
Metrics: F1 score ROUGE score similarity score
Baselines: KIVI-2 KIVI-4 GEAR FP16
Builds on: KIVI GEAR Matching Pursuit
Closest papers by meaning
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.90)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.90)
- Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries (ICML 2025 · similarity 0.89)
- SALS: Sparse Attention in Latent Space for KV Cache Compression (NeurIPS 2025 · similarity 0.89)
- Value-Guided KV Compression for LLMs via Approximated CUR Decomposition (NeurIPS 2025 · similarity 0.89)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.