Research map / Large language models / Reasoning and chain of thought
SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SmallKV, which uses a small same-family LLM's attention scores to compensate for KV cache eviction in a larger LLM, covering both saliency shift and marginal tokens, and reports accuracy gains and 1.75-2.56x throughput gains.
Problem
Token-level KV cache eviction is irreversible, so it fails when token saliency shifts during decoding and it over-compresses marginal tokens that still matter for output quality.
Approach
Attention patterns of same-family LLMs of different scales are highly similar, so a small language model (SLM) keeps a full KV cache and its attention scores are used to (1) guide eviction in the large model, recovering tokens whose saliency changes, and (2) approximate attention for marginal tokens by using SLM scores with only their V cache retained.
Key results
- On LongBench at 5% KV budget with Qwen2-0.5B assisting Qwen2-7B, SmallKV keeps average scores near full cache (e.g., Multi-Doc QA 37.54 vs H2O 24.14 and full 40.78).
- With 5% KV budget on BBH/GSM8K/MT-Bench-style benchmarks, H2O and PyramidInfer drop from 79.4 (full) to 36.7 and 47.0 while SmallKV holds 73.0.
- Efficiency on one A100 with Qwen2-7B: at batch 64 and 2048+256 length, SmallKV (20% budget) reaches 195.50 tokens/s (3.14x over eager Accelerate) versus 76.32 for H2O.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SmallKV instead of H₂O | Qwen2-7B KV cache compression · LongBench τ=0.1, Multi-Doc QA AVG | score: 32.51 -> 40.84 | Helped |
| Use SmallKV instead of H₂O | Qwen2-7B KV cache compression · Multi-user concurrency, batch 64, 2048+256 | throughput (tokens/s): 76.32 (1.23x) -> 195.50 (3.14x) | Helped |
| Use SmallKV instead of H₂O | Qwen2-7B KV cache compression · Extremely long context, batch 4, 16384+512 | throughput (tokens/s): 689.07 (1.45x) -> 1203.42 (2.54x) | Helped |
| Add marginal information compensation (ablation) | SmallKV · BBH | performance | Helped |
| Add saliency shift compensation (ablation) | SmallKV without marginal information compensation · BBH | performance | Helped |
| Increase SLM scale from Qwen2.5-0.5B to Qwen2.5-1.5B (ablation) | SmallKV with Qwen2.5-14B · BBH | performance: 4.9% | Helped |
| Increase SLM scale from Qwen2.5-0.5B to Qwen2.5-3B (ablation) | SmallKV with Qwen2.5-14B · BBH | performance: 6.8% | Helped |
| Increase SLM scale from Qwen2.5-0.5B to Qwen2.5-7B (ablation) | SmallKV with Qwen2.5-14B · BBH | performance: 8.5% | Helped |
| Increase SLM scale from Qwen2.5-0.5B to Qwen2.5-1.5B (ablation) | SmallKV with Qwen2.5-14B · BBH, 5% KV cache budget | performance: 30.2% | Helped |
| Increase SLM scale from Qwen2.5-0.5B to Qwen2.5-3B (ablation) | SmallKV with Qwen2.5-14B · BBH, 5% KV cache budget | performance: 63.8% | Helped |
| Increase SLM scale from Qwen2.5-0.5B to Qwen2.5-7B (ablation) | SmallKV with Qwen2.5-14B · BBH, 5% KV cache budget | performance: 84.4% | Helped |
| Compress SLM KV cache to 20% instead of full (ablation) | SmallKV with Qwen2-0.5B assisting Qwen2-7B · BBH | accuracy: 0.4359 -> 0.2873 | Hurt |
Surprising findings
- Attention similarity between Qwen2-7B and Qwen2-0.5B (0.9622 cosine) is much higher than between Qwen2-7B and Llama3-1B (0.6917), suggesting similarity is mainly within a model series.
- Scaling the SLM from 0.5B to 7B yields larger gains at low KV budgets (e.g., 84.4% improvement at 5% budget) than at high budgets.
Limitations
- Attention similarity between LLM and SLM is not guaranteed precisely, so the method is not lossless.
- No formal theoretical proof of the cross-scale attention similarity; the explanation is speculative.
- The SLM adds memory and compute overhead that cannot be ignored when SmallKV runs without speculative decoding.
Experimental setup
Datasets: Wikitext-v2
Benchmarks: GSM8K BBH MT-Bench LongBench MMLU Wikitext-v2
Metrics: accuracy score TPOT (ms) TTFT (s) end-to-end throughput (tokens/s) Jaccard similarity cosine similarity
Baselines: H2O PyramidInfer Hugging Face Accelerate (eager) Hugging Face Accelerate (FlashAttention SDPA)
Builds on: H2O PyramidInfer speculative decoding FlashAttention
Closest papers by meaning
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.92)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.92)
- EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction (NeurIPS 2026 · similarity 0.92)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.91)
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.91)
- AttentionPredictor: Temporal Patterns Matter for KV Cache Compression (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.