Research map / Architectures and efficiency / Attention and transformers
CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAKE, a KV cache eviction method that allocates per-layer cache budgets from spatial attention entropy and temporal attention variance, applies cascading layer-wise prefill management, and uses a mean-plus-variance eviction indicator.
Problem
KV cache eviction under fixed memory budgets typically assigns uniform cache sizes across layers, ignoring that layers have different spatial and temporal attention patterns, which hurts performance under tight memory.
Approach
Measure each layer's cache preference via the product of attention entropy (spatial dispersion) and column variance (temporal shift) over a recent observation window, normalize these scores into per-layer budgets, and perform budget-constrained eviction in a cascading manner during prefill so peak memory stays bounded; the eviction indicator adds mean and variance of attention per token.
Key results
- With a 128L total budget (about 3.2% of cache) on Llama2-7B-Chat, CAKE averages 29.29 on LongBench versus 28.20 for PyramidKV and 28.09 for SnapKV.
- CAKE reports about 48.63% peak memory reduction versus full cache at 128K context on Mistral-7B-Instruct-v0.3 and over 10x decoding speedup at 128K.
- On Llama3.1-8B-Instruct at 128L, CAKE averages 43.39 versus 42.53 for the next best (SnapKV).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use preference-prioritized adaptive cache allocation instead of the strongest reported baseline | CAKE on Llama2-7B-Chat · LongBench, 128L | average score: 28.20 -> 29.29 | Helped |
| use preference-prioritized adaptive cache allocation instead of the strongest reported baseline | CAKE on Llama2-7B-Chat · LongBench, 1024L | average score: 32.55 -> 32.98 | Helped |
| use preference-prioritized adaptive cache allocation instead of the strongest reported baseline | CAKE on Llama3.1-8B-Instruct · LongBench, 128L | average score: 42.53 -> 43.39 | Helped |
| use preference-prioritized adaptive cache allocation instead of the strongest reported baseline | CAKE on Llama3.1-8B-Instruct · LongBench, 1024L | average score: 47.95 -> 48.13 | Helped |
| use preference-prioritized adaptive cache allocation instead of the strongest reported baseline | CAKE on Mistral-7B-Instruct-v0.3 · LongBench, 128L | average score: 35.81 -> 37.33 | Helped |
| use preference-prioritized adaptive cache allocation instead of the strongest reported baseline | CAKE on Mistral-7B-Instruct-v0.3 · LongBench, 1024L | average score: 42.18 -> 42.69 | Helped |
| use CAKE instead of full cache | Mistral-7B-Instruct-v0.3 with FlashAttention-2 · 128K context | decoding speedup: over 10× | Helped |
| use CAKE instead of full cache | Mistral-7B-Instruct-v0.3 with FlashAttention-2 · 128K context | peak memory usage reduction: 48.63% | Helped |
| add preference-prioritized adaptive allocation rather than uniform, pyramid-shaped, or random allocation (ablation) | CAKE on Llama2-7B-Chat · LongBench, 128L | performance | Helped |
| add the mean-plus-variance eviction indicator rather than mean-only, variance-only, or multiplicative combination (ablation) | CAKE on Llama2-7B-Chat · LongBench, 128L | performance | Helped |
| use mean attention instead of variance-only token scoring (ablation) | CAKE on Llama2-7B-Chat · LongBench, 128L | performance | Helped |
| use a multiplicative mean-and-variance indicator instead of an individual metric (ablation) | CAKE on Llama2-7B-Chat · LongBench, 128L | performance | Hurt |
Surprising findings
- CAKE slightly surpasses full cache on NeedleBench single-needle retrieval, which the authors attribute to pruning removing distracting details.
Limitations
- Operates at layer granularity and does not capture head-level attention variation.
- Temperature hyperparameters tau1 and tau2 and gamma are tuned via grid search, adding configuration burden.
Experimental setup
Datasets: LongBench
Benchmarks: LongBench NeedleBench
Metrics: average score peak memory decoding latency
Baselines: StreamingLLM H2O TOVA SnapKV PyramidKV
Builds on: SnapKV PyramidKV H2O FlashAttention-2
Closest papers by meaning
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.93)
- Taming the Fragility of KV Cache Eviction in LLM Inference (ICLR 2026 · similarity 0.91)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.91)
- SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs (ICML 2025 · similarity 0.91)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.91)
- DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.