Research map / Large language models / Reasoning and chain of thought

Taming the Fragility of KV Cache Eviction in LLM Inference

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes defensive aggregation (worst-case max plus adaptive prior correction) replacing mean aggregation in KV cache eviction, yielding DefensiveKV and Layer-DefensiveKV that lose far less generation quality at 20% cache.

Problem

KV cache eviction methods rely on a stability assumption and mean aggregation of importance scores, which is fragile: importance can shift abruptly, causing large worst-case quality losses under heavy compression.

Approach

Replace mean aggregation with defensive aggregation: estimate each entry's worst-case risk as the max importance over historical queries, then apply an adaptive prior-risk floor (head-average of worst-case risks) to correct under-observation; add layer-wise joint selection for Layer-DefensiveKV.

Key results

• On Llama-3.1-8B at 20% cache, LongBench quality loss is 4.8% for DefensiveKV and 2.6% for Layer-DefensiveKV versus 11.1% for CriticalKV (about 2.3x and 4.3x reductions) per the introduction; the Section 4 text reports 5.1% and 2.3%.
• Worst-case retained importance rises from 0.33 (mean aggregation) to 0.61 with defensive aggregation on a Llama-3.1-8B Government Report trial.
• At 10% cache on 32K NIAH, DefensiveKV scores 139 on Mistral-7B versus CriticalKV's 28, and 194 on Llama-3.1-8B versus CriticalKV's 140.

Surprising findings

• Defensive aggregation applied to AdaKV raises its LongBench average from 41.39 to 44.97 without other changes, showing the aggregation step generalizes across eviction methods.
• Ablation on the adaptive prior correction: fixed correction thresholds (1E-3, 1E-5) fail to beat the no-correction variant, while the adaptive version improves the NIAH score from 179 to 194.

Limitations

• Gains are evaluated mainly on LongBench, RULER and NIAH with three models; full RULER evaluation was only run for Llama-3.1-8B at 20% and 40%.
• The worst-case estimate uses only a limited observation window (32 tokens), so rare future risks may still be missed, as the authors acknowledge via the prior correction.
• DuoAttention is N/A for Qwen2.5-32B, so comparisons are incomplete for that model.

Experimental setup

Datasets: LongBench RULER Needle-in-a-Haystack (Paul Graham essays)

Benchmarks: LongBench RULER Needle-in-a-Haystack

Metrics: task score quality loss relative to full cache worst-case retained importance TTFT decoding latency peak memory throughput

Baselines: StreamingLLM SnapKV AdaKV CAKE CriticalKV DuoAttention

Builds on: CriticalKV SnapKV AdaKV CAKE StreamingLLM

Closest papers by meaning

• CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.91)
• Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.91)
• DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction (NeurIPS 2026 · similarity 0.91)
• Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective (ICLR 2026 · similarity 0.91)
• Learning to Evict from Key-Value Cache (ICML 2026 · similarity 0.90)
• SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.90)