Research map / Large language models / Language, safety and interpretability
Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free KV cache compression that merges adjacent keys (Newton-style optimal merged key) and losslessly merges values by summation with cardinality normalization, exploiting local key homogeneity and value heterogeneity.
Problem
Long-context LLM inference is limited by quadratic attention compute and linearly growing KV cache storage; existing cache merging methods treat keys and values identically despite their different local distributions.
Approach
Observes that adjacent keys receive similar attention (local homogeneity) while adjacent values are heterogeneous; merges adjacent keys into one optimal key via a curvature-based (Fisher-approximated Hessian) Newton-like solution, and losslessly merges values by addition with a cardinality vector that preserves the softmax denominator.
Key results
- On LLaMA3.1-8B-Instruct, AsymKV reaches 43.95 average on LongBench vs 38.89 for H2O.
- On TopicRet with LLaMA3.1-8B, AsymKV scores 75.33 vs 80.00 for full context, far above context-segmentation and prompt compression baselines.
- AsymKV matches full-context performance on LongBench v2 overall (30.02 vs 30.02) while outperforming H2O (28.23).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AsymKV instead of H₂O | Llama3.1-8B-Instruct · LongBench | average score: 38.89 -> 43.95 | Helped |
| Use AsymKV instead of Full Context | Llama2-7B-chat · LongBench | average score: 32.00 -> 33.12 | Helped |
| Use AsymKV instead of Full Context | Mistral-7B-Instruct-v0.3 · LongBench | average score: 46.40 -> 41.33 | Hurt |
| Use AsymKV instead of Full Context | Qwen2-7B-Instruct · LongBench | average score: 43.81 -> 38.76 | Hurt |
| Use AsymKV instead of Full Context | Llama3.1-8B-Instruct · LongBenchV2 | Overall: 30.02 -> 30.02 | No clear effect |
| Use AsymKV instead of PyramidKV | Llama3-8B-Instruct · LongBench | average score: 41.49 -> 42.32 | Helped |
| Use the theoretically derived optimal key merge strategy instead of alternative key merge strategies (ablation) | AsymKV · MuSiQue | score | Helped |
| Use asymmetric cardinality-normalized value merging instead of the same merge strategy for keys and values (ablation) | AsymKV · MuSiQue | score | Helped |
Surprising findings
- AsymKV often outperforms the full-context model on LongBench, which the authors attribute to a regularization effect from merging low-attention tokens.
- Gradient terms in the Newton update were found to exceed Hessian terms by about six orders of magnitude, so the authors dropped them and used only curvature information.
Limitations
- Compatibility with FlashAttention and vLLM not yet addressed, described as an engineering problem.
- Relies on a Fisher-information diagonal approximation of the Hessian and drops gradient terms for stability, which is heuristic.
- Experiments are single-run with no error bars.
Experimental setup
Datasets: ShareGPT QASPER MultiFieldQA
Benchmarks: LongBench LongBench v2 TopicRet (L-Eval) HotpotQA MuSiQue
Metrics: average score accuracy peak GPU memory generation speed
Baselines: H2O CaM StreamingLLM LongCache LLMLingua-2 SnapKV PyramidKV TOVA D2O L2-Norm
Builds on: H2O CaM StreamingLLM LongCache
Closest papers by meaning
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.91)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.91)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.90)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.90)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.90)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.