Research map / Architectures and efficiency / Attention and transformers
Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ada-KV, a head-wise adaptive budget allocation for top-k KV cache eviction that is plug-and-play with SnapKV and Pyramid, evaluated on Ruler and LongBench under question-aware and question-agnostic settings.
Problem
KV cache eviction methods allocate budgets uniformly across attention heads, ignoring heads' differing attention concentration, which wastes budget and increases eviction loss for long-context LLM inference.
Approach
Derive an upper bound on L1 eviction loss between pre- and post-eviction attention outputs, show top-k eviction minimizes it for a given allocation, and allocate per-head budgets by globally selecting the top-B attention weights across heads and counting selections per head (with a safeguard alpha=0.2).
Key results
- On Ruler 16K with Llama-3.1-8B-Instruct question-agnostic, Ada-SnapKV raises SnapKV's score at 20% cache from 44.02 to 53.29 and at 80% from 87.59 to 92.67.
- On LongBench question-aware with Llama-3.1-8B-Instruct at budget 2048, Ada-SnapKV reaches 49.15 versus SnapKV's 49.09, approaching full-cache 49.20.
- Ada-SnapKV's peak memory and decoding latency at budget 1024 are comparable to SnapKV while both beat full cache.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive head-wise budget allocation to SnapKV | SnapKV · Ruler 16K, question-agnostic, 80% cache | average score: 87.59 -> 92.67 | Helped |
| Add adaptive head-wise budget allocation to SnapKV | SnapKV · Ruler 16K, question-agnostic, 20% cache | average score: 44.02 -> 53.29 | Helped |
| Add adaptive head-wise budget allocation to SnapKV | SnapKV · Ruler 16K, question-agnostic, 80% cache, S-NIAH-3 | score: 62.4 -> 97.6 | Helped |
| Add adaptive head-wise budget allocation to SnapKV | SnapKV · Ruler 16K, question-agnostic, 80% cache, MK-NIAN-2 | score: 85.2 -> 99.6 | Helped |
| Add adaptive head-wise budget allocation to SnapKV | SnapKV · LongBench, question-aware, budget 2048 | average score: 49.09 -> 49.15 | Helped |
| Add adaptive head-wise budget allocation to SnapKV | SnapKV · LongBench, question-agnostic, 20% cache, average | average score: 41.29 -> 42.87 | Helped |
| Add adaptive head-wise budget allocation to SnapKV | SnapKV · LongBench, question-agnostic, 40% cache, average | average score: 45.52 -> 46.24 | Helped |
| Add adaptive head-wise budget allocation to SnapKV | SnapKV · LongBench, question-agnostic, 10% cache, Synthetic | score: 36.50 -> 34.88 | Hurt |
| Set safeguard α to 0 instead of 0.2 (ablation) | Ada-SnapKV · LongBench, question-aware, budget 128 | average score: 35.48 -> 35.69 | Helped |
| Set safeguard α to 0 instead of 0.2 (ablation) | Ada-SnapKV · LongBench, question-aware, budget 1024 | average score: 41.61 -> 41.44 | Hurt |
| Increase safeguard α from 0.2 to 0.5 (ablation) | Ada-SnapKV · LongBench, question-aware, budgets 128, 256, 512, and 1024 | average score: 35.48 -> 35.2; 38.61 -> 38.46; 40.58 -> 40.66; 41.61 -> 41.62 | Mixed |
| Transfer compression from question-aware to question-agnostic | SnapKV · LongBench, Llama-3.1-8B-Instruct, budget 2048 | average score: 49.09 -> 42.86 | Hurt |
Surprising findings
- Question-agnostic compression causes large performance drops for all Top-k methods, e.g. SnapKV drops from 49.09 to 42.86 on LongBench at budget 2048 with Llama.
- Code tasks are largely insensitive to cache compression and sometimes improve after compression.
Limitations
- Head-wise allocation is restricted to within a single layer; cross-layer allocation is future work.
- The safeguard alpha is a fixed hyperparameter (0.2) rather than tuned per model or budget.
- The theoretical bound is derived with window size 1 and the L1 loss, with the bound's tightness not guaranteed to translate to quality.
Experimental setup
Datasets: Ruler LongBench
Benchmarks: Ruler LongBench
Metrics: average score peak memory decoding latency
Baselines: SnapKV Pyramid StreamingLLM DuoAttention HeadKV
Builds on: SnapKV PyramidKV StreamingLLM H2O FlashAttention-2
Closest papers by meaning
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.94)
- CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.93)
- Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.92)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.92)
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.92)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.