Research map / Large language models / Reasoning and chain of thought
KV cache compression
Evict or merge key-value cache entries to cut memory in LLM inference.
45 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2024–2026) · an approach in Reasoning and chain of thought, keyword cluster “decoding · speculative · cache”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
5 papers in 2023–24 and 40 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 3.8% to 2.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2024 · ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describe (CVPR 2024 · 108 citations)
- 2025 · KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse (NeurIPS 2025 · 4 citations)
- 2026 · TriAttention: Efficient Long Reasoning with Trigonometric KV Compression (ICML 2026 · 0 citations)
Earliest papers here
- ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describe (CVPR 2024 · 108 citations)
- SnapKV: LLM Knows What You are Looking for Before Generation (NeurIPS 2024 · 51 citations)
- DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving (ICML 2024 · 1 citation)
What has been tried
15 papers here report 169 measured changes (a component added, removed or swapped, compared with a baseline): helped 66%, no clear effect 7%, hurt 26%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add appearance evolution to the pure encoder | GOT-10k test · AO | 71.0 -> 74.2 | Helped | ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describe (CVPR 2024 · 108 citations) |
| Add pooling-based clustering to selected attention positions | LongEval-Lines · retrieval accuracy | Helped | SnapKV: LLM Knows What You are Looking for Before Generation (NeurIPS 2024 · 51 citations) | |
| Add 5 link tokens instead of using KVLink0 | HotpotQA · accuracy | 48.1% -> 55.6% | Helped | KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse (NeurIPS 2025 · 4 citations) |
| Recompute 2 initial tokens per chunk rather than use CacheBlend-15 | Six datasets, synchronous workload · TTFT | up to 300% | Helped | EPIC: Efficient Position-Independent Caching for Serving Large Language Models (ICML 2025 · 2 citations) |
| use redundancy-only selection (λ=0) instead of balanced importance and redundancy | MATH-500 · accuracy | Hurt | R-KV: Redundancy-aware KV Cache Compression for Reasoning Models (NeurIPS 2025 · 1 citation) | |
| Raise ξs to 500ms instead of calibrating it to the target tail | WildChat, capacity 10000 · requests exceeding 200ms | -19.3% | Hurt | Tail-Optimized Caching for LLM Inference (NeurIPS 2025 · 1 citation) |
| increase overlap O from 0 to S/2 | Long-context understanding, QA tasks · score | 19.48 -> 18.48 | Hurt | LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models (ICML 2025 · 0 citations) |
| Add marginal information compensation | BBH · performance | Helped | SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference (NeurIPS 2025 · 0 citations) | |
| Use selector window R=32 instead of R=1, R=8, or R=128 | AIME 2024 · pass@1 | 49.2, 48.3, and 49.2 -> 52.9 | Helped | Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoning (NeurIPS 2025 · 0 citations) |
| vary scoring chunk size | SCBench; LLaMA3.1-8B; 0.3 KV cache ratio · average performance difference | below 2% | No clear effect | KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction (NeurIPS 2025 · 0 citations) |
| Vary the anchor vector choice | Ablation benchmark results · Accuracy | No clear effect | KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments (NeurIPS 2025 · 0 citations) | |
| Use even context partitioning rather than searched context partitioning | Falcon 7B, 4k context · TTFT | Hurt | KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation (ICML 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 40 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- GSM8K · 10 of 40 papers
- MATH500 · 8 of 40 papers
- LongBench · 6 of 40 papers
- RULER · 5 of 40 papers
- HotpotQA · 5 of 40 papers
- AIME 2024 · 5 of 40 papers
- MMLU · 4 of 40 papers
- Needle-In-A-Haystack · 3 of 40 papers
Baselines
- SnapKV · 12 of 40 papers
- H2O · 11 of 40 papers
- R-KV · 7 of 40 papers
- StreamingLLM · 6 of 40 papers
- DuoAttention · 5 of 40 papers
- TOVA · 4 of 40 papers
- PyramidInfer · 3 of 40 papers
- CacheBlend · 3 of 40 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 45 papers; from OpenAlex and the papers' affiliations).
- Microsoft · 3 papers
- Princeton University · 3 papers
- Seoul National University · 3 papers
- Georgia Institute of Technology · 2 papers
- NVIDIA · 2 papers
- Google · 2 papers
- Nanjing University · 2 papers
- Huawei · 2 papers
- NAVER · 2 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient LLM serving (126 papers)
- Speculative decoding (92 papers)
Elsewhere in the topic
- RL with verifiable rewards for reasoning (270 papers)
- Efficient and adaptive thinking (231 papers)
- Test-time compute scaling (205 papers)
- Long CoT dynamics and efficiency (136 papers)
- Routing, pruning and distillation for reasoning (135 papers)
- Chain-of-thought training and analysis (127 papers)
- Search and self-correction in reasoning (123 papers)
- Process reward models (39 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).