Research map / Large language models / Reasoning and chain of thought
CaM: Cache Merging for Memory-efficient LLMs Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cache Merging (CaM), which merges to-be-evicted KV cache value vectors into retained local tokens using a Bernoulli sampling mask based on accumulated attention, to reduce output perturbation of KV cache eviction methods.
Problem
KV cache eviction for memory-efficient LLM inference causes output perturbation that grows with compression ratio, degrading performance on QA, summarization and language modeling.
Approach
Instead of discarding evicted KV value caches, merge them evenly into a window of local tokens' value caches, with a probabilistic merge mask proportional to the cumulative attention of the evicted token relative to the local window average; this provably reduces output perturbation relative to eviction under a condition on attention ratios.
Key results
- On LLaMA-7B with a 20% KV cache budget, CaM improves StreamingLLM on RTE from 49.5 to 54.6 zero-shot accuracy.
- CaM improves H2O on LLaMA-7B under a 20% budget by 4.8%, 3.3%, and 9.0% on OpenBookQA, PiQA, and COPA respectively.
- On an NVIDIA 3090, applying CaM to StreamingLLM changes generation latency from 26.1 to 27.3 ms/token and GPU memory from 19.1 GB to 19.3 GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CaM to Scissorhands | LLaMA-7B · OpenBookQA | accuracy: 22.0 -> 25.4±0.2 | Helped |
| Add CaM to StreamingLLM | LLaMA-7B · PiQA | accuracy: 67.6 -> 69.8±0.2 | Helped |
| Add CaM to StreamingLLM | LLaMA-7B · Winogrande | accuracy: 52.7 -> 54.5±0.1 | Helped |
| Add CaM to StreamingLLM | LLaMA-7B · COPA | accuracy: 64.0 -> 68.0±0.0 | Helped |
| Add CaM to Scissorhands | LLaMA-7B · RTE | accuracy: 51.6 -> 53.4±0.1 | Helped |
| Add CaM to H2O | LLaMA-7B · MathQA | accuracy: 22.1 -> 22.4±0.0 | Helped |
| Use average merging across local tokens instead of merging into a randomly selected local token (ablation) | CaM on LLaMA-7B · OpenBookQA | accuracy: 30.4±0.3 -> 31.8±0.2 | Helped |
| Add the merge mask (ablation) | CaM on LLaMA-7B · COPA | accuracy: 63.0±0.0 -> 68.0±0.0 | Helped |
| Use accumulated attention rather than current-step attention to derive the merge mask (ablation) | CaM on LLaMA-7B · RTE | accuracy: 54.5±0.2 -> 54.6±0.2 | No clear effect |
| Add CaM to StreamingLLM | PG-19 test set, first sample · PG-19 test set, first sample | throughput: 26.1 ms/token -> 27.3 ms/token | Hurt |
| Add CaM to StreamingLLM | PG-19 test set, first sample · PG-19 test set, first sample | GPU memory: 19.1GB -> 19.3GB | Hurt |
Surprising findings
- Merging all evicted tokens indiscriminately (without the merge mask) degrades performance below the StreamingLLM baseline on some tasks.
- Language modeling shows smaller gains from CaM because StreamingLLM already performs well there.
Limitations
- Merge token selection and merge ratio are heuristic (uniform averaging over a local window); the authors call for better selection.
- Layer-wise adaptivity of merging is not explored; the same policy is applied across all layers.
- Gains are smaller on language modeling tasks where the baseline is already strong.
Experimental setup
Datasets: COPA MathQA OpenBookQA PiQA RTE Winogrande XSUM CNN/DailyMail Wikitext-2 PG-19
Benchmarks: OpenBookQA PiQA Winogrande COPA RTE MathQA XSUM CNN/DailyMail Wikitext-2 PG-19
Metrics: accuracy Rouge-1 coverage perplexity generation throughput GPU memory
Baselines: StreamingLLM H2O Scissorhands Full cache
Builds on: StreamingLLM H2O Scissorhands Token Merge (ToMe)
Closest papers by meaning
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.91)
- Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.91)
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.91)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.91)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.90)
- ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.