Research map / Architectures and efficiency / Attention and transformers
CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies attention heads with sequential consistency between prefilling and decoding and proposes CateKV, a hybrid KV cache that keeps few critical tokens for consistent heads and most tokens for adaptive heads, classified by a coefficient-of-variation score.
Problem
Long-context LLM inference is limited by KV cache memory and decoding latency; existing eviction methods lose accuracy and retrieval methods do not reduce memory.
Approach
A coefficient-of-variation score over binarized observation-window attention from prefilling classifies heads as consistent or adaptive; a static per-model head map from a reference dataset then assigns a small chunk budget to consistent heads and a large retention ratio to adaptive heads.
Key results
- On RULER-128K with Llama-3-8B-1M, CateKV (r=0.3, eta=0.7) reduces memory by 2.72x and single-sample decoding latency by 2.18x with -0.25 accuracy change on RULER.
- In batch inference on Llama-3-8B-1M, CateKV raises throughput to 909.69 (3.96x) versus full attention's 229.37.
- On LongBench with Llama-3-8B-1M, CateKV keeps 42% of KV cache with average 31.48 vs full 31.27.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CateKV instead of full attention with r=0.3 and η=0.7 | Llama-3-8B-1M · RULER-128K | Average accuracy: 84.10 -> 83.85 | Hurt |
| Use CateKV instead of full attention with r=0.3 and η=0.7 | Llama-3-8B-1M · LongBench | Average performance: 31.27 -> 31.21 | No clear effect |
| Use CateKV with r=0.3 and η=0.7 rather than full attention | Llama-3-8B-1M · Single-sample inference | Latency: 54.28 -> 24.86; $2.18× | Helped |
| Use CateKV with r=0.3 and η=0.7 rather than full attention | Llama-3-8B-1M · Batch-sample inference | Throughput: 229.37 -> 909.69; $3.96× | Helped |
| Add CateKV to Duoattention by replacing part of streaming heads with consistent heads | Duoattention on Llama-3-8B-1M · RULER 8K–256K | Avg. accuracy: 68.46 -> 87.26 | Helped |
| Add CateKV to MInference | MInference on Llama-3-8B-1M · RULER 8K–256K | Avg. accuracy: 87.32 -> 87.74 | Helped |
| Reduce the adaptive-head ratio to 0.1 (ablation) | CateKV · RULER-128K | Accuracy | No clear effect |
| Reduce the adaptive-head ratio below approximately 0.2 (ablation) | CateKV · RULER-128K | Accuracy | Hurt |
| Reduce adaptive-head retention ratio below approximately 0.6 (ablation) | CateKV · RULER-128K | Accuracy | Hurt |
| Reduce the consistent-head sparse budget to approximately 0.78% (1024) (ablation) | CateKV with r=0.4 and η=1.0 · RULER-128K | Average accuracy | No clear effect |
| Reduce the consistent-head sparse budget (ablation) | CateKV with r=0.4 and η=1.0 · RULER-128K | Memory usage and latency | No clear effect |
| Set adaptive-head ratio to 0.3 instead of 0.4 (ablation) | CateKV on Phi-3-Mini-128K · RULER-128K | Average accuracy: 71.69 -> 71.48 | Hurt |
Surprising findings
- Retaining only a small fraction of tokens (1024-2048 budget) for consistent heads keeps accuracy near full attention, suggesting global information relies mainly on adaptive heads.
- Combining CateKV with MInference prefilling acceleration yields slight accuracy improvement on RULER (87.74 vs 87.32 average).
Limitations
- Head classification relies on a fixed reference dataset and hyperparameters (k, alpha, r, eta) tuned per model.
- Accuracy degrades noticeably when the adaptive head ratio falls below about 0.2 or retention ratio below about 0.6.
- Theoretical analysis (Rademacher bound, retention accuracy lower bound) is loosely connected to empirical results.
Experimental setup
Datasets: RULER Variable Tracking (reference set, 100 samples at 128K) WikiText
Benchmarks: RULER LongBench Needle in a Haystack Full LongBench
Metrics: accuracy memory usage decoding latency throughput batch size
Baselines: StreamingLLM SnapKV PyramidKV DuoAttention Quest ShadowKV MInference
Builds on: StreamingLLM SnapKV DuoAttention Quest ShadowKV
Closest papers by meaning
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.93)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.92)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.92)
- SnapKV: LLM Knows What You are Looking for Before Generation (NeurIPS 2024 · similarity 0.92)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.92)
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.