Research map / Architectures and efficiency / Attention and transformers

Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs

ICLR 2024 · Architectures and efficiency › Attention and transformers · approach: KV cache eviction and compression · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FastGen, a training-free adaptive KV cache compression method that profiles each attention head during prompt encoding and assigns it a compression policy (special tokens, punctuation, locality, frequency, or full cache) to cut KV memory while keeping generation quality.

Problem

The KV cache of autoregressive LLMs grows linearly with generation length and consumes large GPU memory; existing compression methods either need retraining or apply one fixed eviction policy to all heads, ignoring that attention heads have different structures.

Approach

Profile each attention head's attention map once during prompt encoding, choose the minimum-memory combination of compression policies (special tokens, punctuation, frequency, locality, full) that recovers a target fraction of attention scores, and build a head-specific KV cache without any fine-tuning.

Key results

• On AlpacaEval, FastGen reaches a 45% win rate with 44.9% pruned KV cache on Llama 1-65B versus 16.9% on Llama 1-7B.
• Across 7B-65B models, FastGen achieves about 20% memory reduction at 7B/13B and about 40% at 65B while keeping win rate above 45%.
• End-to-end latency vs Hugging Face Accelerate on Llama 1-7B on V100 is reduced by 16% to 55% depending on batch size and generation length.

Measured effects reported in the paper

Surprising findings

• Fixed (non-adaptive) KV policies such as local+frequent (H2O-style) degrade quality more than adaptive FastGen at equal cache budgets.
• Attention structure per head is stable across decoding steps, so a one-shot profile from the prompt suffices.

Limitations

• Evaluated only on Llama 1 models with multi-head attention; grouped-query attention not explored.
• Relies on GPT-4 pairwise win rate against full cache, with a 45% threshold defining 'no loss'.
• Custom sparsity kernel is not fully optimized; latency gains vs DeepSpeed are smaller.

Experimental setup

Datasets: LIMA Open Assistant (oasst1) GSM8K HumanEval Natural Questions TriviaQA AlpacaEval

Benchmarks: GSM8K HumanEval NQ TQA AlpacaEval

Metrics: F1 Pass@1 win rate vs full cache pruned KV cache ratio end-to-end latency GPU memory

Baselines: H2O (local+frequent) Scissorhands Frequency-only KV cache Local-only KV cache Full KV cache DeepSpeed Inference Hugging Face Accelerate

Builds on: H2O Scissorhands DeepSpeed Inference Hugging Face Accelerate

Closest papers by meaning

• RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.94)
• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.92)
• Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.92)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.92)
• EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction (NeurIPS 2026 · similarity 0.92)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.92)