Research map / Architectures and efficiency / Attention and transformers
Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MorphKV, an inference-time KV cache eviction method that keeps a constant-sized cache by retaining distant tokens ranked by attention from a recent window, evaluated on long-response and long-context benchmarks.
Problem
KV cache grows linearly with context and response length, causing memory and bandwidth bottlenecks; existing compression methods either drop distant context or lose accuracy, especially for long-response generation.
Approach
Keep the last R tokens plus the top-C older tokens ranked by fused (sum or max) attention scores that recent window tokens assign to them, updating the cache each step so its size stays at C+R; the method is GQA-aware, aggregating attention across grouped query heads.
Key results
- On LongWriter, MorphKV averages about 0.25x full-attention KV cache size, versus 1x for H2O and up to 4x for SnapKV.
- On LongGenBench, MorphKV reaches 45% average accuracy on Llama versus 44% for SnapKV and H2O, and uses about 0.55x full-attention memory versus 5.01x peak for SnapKV.
- The abstract claims 52.9% memory savings and 18.2% higher accuracy on average over state-of-the-art prior works on long-response tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MorphKV instead of H₂O or SnapKV | LongWriter (en), Llama3.1-8B · LongWriter (en) | Total: 67.7 -> 69.5; 68.5 -> 69.5 | Helped |
| Use MorphKV instead of H₂O or SnapKV | LongWriter (en), Mistral-7B · LongWriter (en) | Total: 81.1 -> 81.1 | No clear effect |
| Use MorphKV instead of SnapKV | LongWriter (en), Phi4 · LongWriter (en) | Total: 63.8 -> 64.65 | Helped |
| Use MorphKV instead of SnapKV | LongWriter (en), Qwen2.5 · LongWriter (en) | Total: 68.4 -> 64.9 | Hurt |
| Use MorphKV instead of H₂O or SnapKV | LongGenBench, Llama · LongGenBench | Average Accuracy: 44 -> 45 | Helped |
| Use MorphKV instead of H₂O or SnapKV | LongGenBench, Mistral · LongGenBench | Average Accuracy: 50 -> 52; 49 -> 52 | Helped |
| Use MorphKV instead of H₂O or SnapKV | LongGenBench, Qwen · LongGenBench | Average Accuracy: 42 -> 47; 39 -> 47 | Helped |
| Use MorphKV instead of SnapKV | LongBench, Mistral-7B · LongBench | Throughput: 1× -> 4.68× | Helped |
| Use max fusion instead of sum fusion (ablation) | MorphKV on LongBench, Llama3.1-8B · LongBench | Average accuracy: +1% | Helped |
| Use sum fusion instead of max fusion (ablation) | MorphKV on LongWriter, most models · LongWriter | Accuracy | Mixed |
| Increase recent window from 32 to 200 tokens (ablation) | MorphKV on LongGenBench, Llama · LongGenBench | Average Accuracy: 42 -> 45 | Helped |
| Increase recent window from 32 to 200 tokens (ablation) | MorphKV on LongGenBench, Mistral · LongGenBench | Average Accuracy: 50 -> 52 | Helped |
Surprising findings
- On LongWriter with Qwen2.5, MorphKV scores below SnapKV (64.9 vs 68.4), which the authors attribute to Qwen's GQA with fewer heads.
- Under very low cache budgets on LongBench, SnapKV-style comparisons show MorphKV's advantage grows, with NarrativeQA gaps reaching up to 88%.
Limitations
- Adds per-step runtime overhead (1.5x vs SnapKV) from computing window attention profiles each step.
- Relies on attention scores to select tokens, requiring partial attention materialization that is not yet fused into FlashAttention.
- Underperforms SnapKV on Qwen2.5 for LongWriter, attributed to fewer KV heads in GQA.
Experimental setup
Datasets: LongWriter LongGenBench LongBench
Benchmarks: LongWriter LongGenBench LongBench
Metrics: LLM judge score accuracy completion rate KV cache size relative to full attention runtime throughput repetition rate
Baselines: SnapKV H2O Full-Attention StreamingLLM Keyformer Scissorhands
Builds on: H2O SnapKV StreamingLLM FlashAttention-2 grouped-query attention
Closest papers by meaning
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.91)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.91)
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.91)
- ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction (NeurIPS 2024 · similarity 0.90)
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.90)
- LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.