Research map / Architectures and efficiency / Attention and transformers
KV cache eviction and compression
Prunes and compresses key-value caches with adaptive budgets to cut LLM inference memory.
92 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in Attention and transformers, keyword cluster “global · local · spatial”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
7 papers in 2023–24 and 85 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 1.2% to 7.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · $S^3$: Increasing GPU Utilization during Generative Inference for Higher Throughput (NeurIPS 2023 · 11 citations)
- 2024 · Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (NeurIPS 2024 · 11 citations)
- 2025 · ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · 3 citations)
- 2026 · Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling (AAAI 2026 · 1 citation)
Earliest papers here
- $S^3$: Increasing GPU Utilization during Generative Inference for Higher Throughput (NeurIPS 2023 · 11 citations)
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (NeurIPS 2024 · 11 citations)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · 5 citations)
What has been tried
37 papers here report 421 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 9%, hurt 22%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add CLA2 using GQA4 | SlimPajama validation · perplexity | 13.36 -> 13.48 | Hurt | Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (NeurIPS 2024 · 11 citations) |
| add the special-token policy to the adaptive policy set | AlpacaEval · win rate | 47.64% -> 49.75% | Helped | Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · 5 citations) |
| Reuse compressed indices across adjacent layers | GSM8K, LLaMA-3-8B-Instruct · accuracy | 74.5 -> 74.6 | No clear effect | ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · 3 citations) |
| use PyramidKV dynamic retrieval-budget allocation | ∞-Bench, 128K context · average accuracy | 49.9 -> 50.1 | Helped | RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (NeurIPS 2025 · 3 citations) |
| Set safeguard α to 0 instead of 0.2 | LongBench, question-aware, budget 128 · average score | 35.48 -> 35.69 | Helped | Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · 3 citations) |
| Use dKV-Cache-Greedy with a local window rather than without a window | GSM8K · accuracy | 67.93 -> 68.23 | Helped | dKV-Cache: The Cache for Diffusion Language Models (NeurIPS 2025 · 2 citations) |
| Add optimization-based retrieval-head identification rather than attention profiling | Mistral-7B-Instruct-v0.2 passkey retrieval · exact match accuracy | Helped | DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads (ICLR 2025 · 2 citations) | |
| Add MInference prefill to ShadowKV | RULER with MInference, Avg. · accuracy | 81.98 -> 82.04 | No clear effect | ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference (ICML 2025 · 1 citation) |
| factorize only keys and values instead of queries, keys, and values | FineWeb-Edu 100B zero-shot evaluation average · accuracy | 51.41 -> 50.04 | Hurt | Tensor Product Attention Is All You Need (NeurIPS 2025 · 1 citation) |
| allocate KV cache sizes adaptively across layers rather than retaining the same size in every layer | MM-Vet, LLaVA-1.5-7B, 30% cache budget · PPL | 14.7 PPL | Helped | PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation (NeurIPS 2025 · 1 citation) |
| Add layer-wise KV-budget reallocation | Mistral-7B, SAMSUM · ROUGE-L | 15.7; 34.01; 37.26; 37.69; 27.41; 26.29; 11.11; 10.48; 8.72; 9.07 | Mixed | SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · 1 citation) |
| Remove query access | SCBench multi-turn mode · Retr.Semantic | 31.4~/~25.7 | Hurt | SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 46 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- LongBench · 17 of 46 papers
- Needle-in-a-Haystack · 10 of 46 papers
- GSM8K · 8 of 46 papers
- PG-19 · 6 of 46 papers
- RULER · 6 of 46 papers
- C4 · 4 of 46 papers
- XSUM · 3 of 46 papers
- CoQA · 3 of 46 papers
Baselines
- StreamingLLM · 21 of 46 papers
- SnapKV · 18 of 46 papers
- H2O · 16 of 46 papers
- Quest · 10 of 46 papers
- PyramidKV · 10 of 46 papers
- KIVI · 7 of 46 papers
- Full Attention · 4 of 46 papers
- TOVA · 4 of 46 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 92 papers; from OpenAlex and the papers' affiliations).
- NVIDIA · 5 papers
- Tsinghua University · 5 papers
- Peking University · 4 papers
- ByteDance · 4 papers
- Microsoft · 3 papers
- Meta · 3 papers
- University of Texas at Austin · 3 papers
- Huawei · 3 papers
- Korea Advanced Institute of Science and Technology · 3 papers
- IBM · 3 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Sparse attention for long-context LLMs (147 papers)
- Block-sparse attention kernels (60 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- State space and recurrent sequence models (172 papers)
- Sparse and structured attention (163 papers)
- Attention head and block redesign (127 papers)
- Efficient and robust vision transformers (124 papers)
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
- Hybrid and interpretable ViT designs (94 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).