Research map / Large language models / Reasoning and chain of thought
DesireKV: Decoupling Sensitivity and Importance for Reasoning-Aware KV Cache Compression
Pengyu Cheng, Jiacheng Wang, Tianle Chen, Bei Liu, Xiaofeng Hou, Jiacheng Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DesireKV, a KV cache compression method for chain-of-thought reasoning that decouples token sensitivity from importance to account for repetitive, low-density reasoning sequences.
Problem
KV cache memory grows large when LLMs produce long chain-of-thought reasoning, and existing KV cache compression methods are suboptimal because reasoning sequences have repetitive logic and low information density.
Approach
Decouples sensitivity and importance of tokens to perform reasoning-aware KV cache compression, rather than applying conventional text-generation compression criteria to reasoning traces.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add outlier-based sensitivity scoring for quantization (vs random 4/8-bit) (ablation) | DesireKV · AIME2024 | Pass@1: 0.4292 -> 0.4625 | Improved |
| add attention-based importance for eviction (vs random eviction) (ablation) | DesireKV · AIME2024 | Pass@1: 0.4375 -> 0.4625 | Improved |
| add confidence-based reasoning protection (ablation) | DesireKV · AIME2024 | Pass@1: 0.4583 -> 0.4625 | Improved |
| add confidence-based reasoning protection (ablation) | DesireKV · GPQA-Diamond | Pass@1: 0.4595 -> 0.4684 | Improved |
| add sensitivity scoring (ablation) | DesireKV · GSM8K | Pass@1: 0.8614 -> 0.8811 | Improved |
| protect top outlier-score tokens instead of random tokens (20% BF16, rest 4-bit) (ablation) | KV quantization · AIME2024 | Pass@1 (%): 37.9 -> 44.6 | Improved |
| replace lowest-attention protected tokens with low-confidence tokens (ablation) | attention-driven protection · AIME2024 | Pass@1 (%): 42.9 -> 44.2 | Improved |
| quantize low-importance tokens (DesireKV) instead of evicting them (RPC) | RPC · AIME2024 | Pass@1: 0.4250 -> 0.5343 | Improved |
| dual importance-sensitivity criteria vs uniform 2-bit quantization of pruned tokens | DDKS · AIME2024 | Pass@1: 0.3375 -> 0.4625 | Improved |
| DesireKV vs K8V4 on average bit width | KIVI K8V4 · four benchmarks, Llama-8B | Ave Bit: 6.0 -> 2.9 | Improved |
| DesireKV vs K8V4 on MATH-500 accuracy | KIVI K8V4 · MATH-500, Llama-8B | Pass@1: 0.8595 -> 0.8595 | No clear effect |
| DesireKV vs K4V4 on GSM8K, Qwen-7B | KIVI K4V4 · GSM8K, Qwen-7B | Pass@1: 0.8795 -> 0.8797 | No clear effect |
Closest papers by meaning
- R-KV: Redundancy-aware KV Cache Compression for Reasoning Models (NeurIPS 2025 · similarity 0.93)
- Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression (ICML 2026 · similarity 0.93)
- BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference (ICML 2026 · similarity 0.92)
- ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models (ICLR 2026 · similarity 0.92)
- Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoning (NeurIPS 2025 · similarity 0.91)
- Value-Aware Stochastic KV Cache Eviction for Reasoning Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
