Research map / Architectures and efficiency / Quantization
KV cache quantization
Compresses key-value caches with vector and mixed-precision quantization for long-context inference.
72 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2024–2026) · an approach in Quantization, keyword cluster “post quantization · ptq · compression”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
11 papers in 2023–24 and 61 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 9.3% to 17.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2024 · KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (NeurIPS 2024 · 51 citations)
- 2025 · QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead (AAAI 2025 · 4 citations)
- 2026 · FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models (AAAI 2026 · 6 citations)
Earliest papers here
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (NeurIPS 2024 · 51 citations)
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (NeurIPS 2024 · 47 citations)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · 14 citations)
What has been tried
30 papers here report 353 measured changes (a component added, removed or swapped, compared with a baseline): helped 66%, no clear effect 12%, hurt 20%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use per-channel quantization for Keys and per-token quantization for Values instead of per-token quantization for both | Wikitext-2 · perplexity | 10.87 -> 7.05; 3.82 perplexity improvement | Helped | KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (NeurIPS 2024 · 51 citations) |
| Use RTN rather than GPTQ for weight quantization | WikiText-2, 4-bit · perplexity | 6.10 -> 8.37 | Hurt | QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (NeurIPS 2024 · 47 citations) |
| Set token retention threshold γ to 0.05 instead of 0 | COQA · score | 0.603 -> 0.647 | Helped | MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · 14 citations) |
| Add channel-separable tokenwise quantization for value cache | GSM8K · accuracy (%) | 52.77 -> 54.74 | Helped | ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification (NeurIPS 2024 · 10 citations) |
| Increase channel coupling from 1 to 4 channels | WikiText-2 test · perplexity | 97.76 -> 5.42 | Helped | KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization (NeurIPS 2024 · 7 citations) |
| Orthogonalize the rows of the JL matrix using QR decomposition | QJL quantization experiments · quantization performance | Helped | QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead (AAAI 2025 · 4 citations) | |
| increase calibration data from 128 to 4096 samples | WikiText-2 · perplexity | 6.994 -> 6.267 | Helped | Extreme Compression of Large Language Models via Additive Quantization (ICML 2024 · 4 citations) |
| Combine subspace linearized PV with STE | WikiText-2 · perplexity | 8.49 -> 8.43 | Helped | PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression (NeurIPS 2024 · 3 citations) |
| increase factor rank from 64 to 256 | WikiText2 test, LLaMa-2 7B · perplexity | 7.37 -> 6.19 | Helped | Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · 3 citations) |
| Add Q,K low-rank compression to GPTQ, without optimal bit allocation | LiveBench · PPL | 38.08 -> 9.10 | Helped | CASP: Compression of Large Multimodal Models Based on Attention Sparsity (CVPR 2025 · 2 citations) |
| Early terminate OMP using a higher reconstruction-error threshold | LongBench average · Average | 50.26 -> 49.63; 50.26 -> 48.90; 50.26 -> 47.73; 50.26 -> 44.33 | Hurt | Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries (ICML 2025 · 1 citation) |
| Increase token batch size | Llama 3.1 8B Instruct on A100-40GB · decompression overhead per token batch | Helped | 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 37 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- WikiText-2 · 13 of 37 papers
- C4 · 12 of 37 papers
- LongBench · 4 of 37 papers
- MMLU · 4 of 37 papers
- WinoGrande · 4 of 37 papers
- HellaSwag · 4 of 37 papers
- PIQA · 4 of 37 papers
- PTB · 4 of 37 papers
Baselines
- GPTQ · 8 of 37 papers
- KIVI · 5 of 37 papers
- ASVD · 5 of 37 papers
- KVQuant · 4 of 37 papers
- AWQ · 4 of 37 papers
- QuaRot · 4 of 37 papers
- KIVI-2 · 4 of 37 papers
- SVD-LLM · 4 of 37 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 72 papers; from OpenAlex and the papers' affiliations).
- Institute of Science and Technology · 3 papers
- Moscow Institute of Physics and Technology · 3 papers
- Yandex · 3 papers
- Cornell University · 3 papers
- Rice University · 3 papers
- Microsoft · 3 papers
- Zhejiang University · 3 papers
- New York University · 2 papers
- Institute of Science and Technology Austria · 2 papers
- Huawei · 2 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Post-training weight quantization (100 papers)
- Post-training quantization of LLMs (136 papers)
- Post-training quantization of ViTs (105 papers)
Elsewhere in the topic
- Mixed-precision quantization (92 papers)
- Low-precision training formats (58 papers)
- Ultra-low-bit LLM quantization (56 papers)
- Binary neural networks (27 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).