Research map / Architectures and efficiency / Attention and transformers
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ChunkKV, a training-free KV cache compression method that keeps semantic chunks instead of individual tokens, plus layer-wise index reuse for lower overhead.
Problem
KV cache memory dominates long-context LLM inference, and token-level eviction methods score tokens in isolation, fragmenting semantic context and degrading performance.
Approach
Group consecutive tokens into chunks as the atomic compression unit, select top-k chunks by aggregated observation-window attention, and reuse the selected indices across groups of adjacent layers because chunk-level indices are more similar across layers.
Key results
- On NIAH with LLaMA-3-8B-Instruct at KV size 128 (8k context), ChunkKV reaches 73.8% vs 65.1% for PyramidKV and 23.7% for StreamingLLM.
- Index reuse with ChunkKV yields up to 20.7% latency reduction and 26.5% throughput improvement over FullKV on A40 with LLaMA-3-8B-Instruct.
- On LongBench at 10% ratio, ChunkKV averages 40.51 for LLaMA-3-8B-Instruct, versus 40.15 for SnapKV and 40.08 for PyramidKV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use semantic chunks instead of isolated-token KV eviction | KV cache compression on GSM8K · GSM8K, 10% KV ratio, DeepSeek-R1-Distill-Llama-8B | accuracy: 62.6% -> 65.7% | Helped |
| Use semantic chunks instead of isolated-token KV eviction | KV cache compression on GSM8K · GSM8K, 10% KV ratio, LLaMA-3.1-8B-Instruct | accuracy: 50.3% -> 65.7% | Helped |
| Use semantic chunks instead of isolated-token KV eviction | KV cache compression on GSM8K · GSM8K, 10% KV ratio, Qwen2-7B-Instruct | accuracy: 64.7% -> 73.5% | Helped |
| Use semantic chunks instead of isolated-token KV eviction | KV cache compression on many-shot GSM8K · Many-shot GSM8K (50-shot), 10% KV ratio, DeepSeek-R1-Distill-Llama-8B | accuracy: 63.2% -> 68.2% | Helped |
| Use semantic chunks instead of isolated-token KV eviction | KV cache compression on JailbreakV · JailbreakV, 10% KV ratio, LLaMA-3.1-8B-Instruct | accuracy: 85.5% -> 87.9% | Helped |
| Use semantic chunks instead of isolated-token KV eviction | KV cache compression on LongBench · LongBench, 10% KV ratio, LLaMA-3-8B-Instruct | performance gap vs FullKV: -3.16% -> -2.29% | Helped |
| Use semantic chunks instead of isolated-token KV eviction | KV cache compression on LongBench-ZH · LongBench-ZH, 10% KV ratio, Qwen2-7B-Instruct | performance gap vs FullKV: -0.64% -> +2.20% | Helped |
| Use semantic chunks instead of isolated-token KV eviction | KV cache compression on NIAH · NIAH, KV cache size 128, LLaMA-3.1-8B-Instruct | accuracy: 65.1% -> 73.8% | Helped |
| Reuse compressed indices across adjacent layers | ChunkKV inference · Input 8192, output 1024 | throughput: 184.08 -> 232.99 | Helped |
| Reuse compressed indices across adjacent layers (ablation) | ChunkKV on GSM8K · GSM8K, LLaMA-3-8B-Instruct | accuracy: 74.5 -> 74.6 | No clear effect |
| Reuse compressed indices across adjacent layers (ablation) | ChunkKV on LongBench · LongBench, LLaMA-3-8B-Instruct | average score: 40.51 -> 40.27 | Hurt |
| Set chunk size to 10 rather than size 3 (ablation) | ChunkKV on LongBench · LongBench, LLaMA-3-8B-Instruct | average score: 40.49 -> 40.51 | Helped |
Surprising findings
- Index reuse with LLaMA-3-8B-Instruct on GSM8K collapses when reusing across 3 or more layers (74.6 to 65.9 at 3 layers, 1.6 at 20 layers), while Qwen2-7B-Instruct is more robust.
Limitations
- Gains over SnapKV and PyramidKV on LongBench are small (about 0.4 points at 10% ratio for LLaMA-3-8B-Instruct).
- Index reuse degrades sharply on GSM8K for LLaMA-3-8B-Instruct beyond two reused layers.
- Chunk size is fixed at 10 for most experiments and only sensitivity-tested on a limited set of values.
Experimental setup
Datasets: LongBench Needle-In-A-HayStack GSM8K JailbreakV
Benchmarks: LongBench LongBench-ZH Needle-In-A-HayStack GSM8K Many-Shot GSM8K JailbreakV
Metrics: accuracy score gap vs FullKV latency throughput Jaccard similarity
Baselines: StreamingLLM H2O SnapKV PyramidKV KIVI FullKV
Builds on: H2O SnapKV StreamingLLM PyramidKV
Closest papers by meaning
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.94)
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.93)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.93)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.93)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.93)
- LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.