Research map / Large language models / Reasoning and chain of thought

Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression

ICML 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces KVFundaBench, a benchmark showing KV cache compression harms high-density reasoning (arithmetic, long-context generation) more than retrieval, and proposes ShotKV, a shot-aware prefill/decoding-separated compression method.

Problem

Existing KV cache compression evaluations focus on retrieval-oriented long-context tasks, masking degradation on high-density reasoning where chain-of-thought coherence and few-shot exemplars matter.

Approach

Benchmark KV compression across world knowledge, commonsense, arithmetic, code, safety, and long-context generation; then propose ShotKV, which treats few-shot exemplars as indivisible semantic units in prefill (attention-ranked shot selection) and applies token-level top-k dynamic compression during decoding.

Key results

• ShotKV reaches 80.37% on many-shot GSM8K at 10% KV ratio versus 51.34% for Random Shot and 82.35% FullKV.
• On LG-GSM8K at 40% compression, ShotKV scores 47.33% vs 46.00% FullKV, while StreamingLLM, H2O, and PyramidInfer drop to 39.50, 32.66, and 38.33.
• ShotKV reduces latency by 11.3% and raises throughput by 13.1% over FullKV at 8192 input and 4096 output tokens on an A40 with batch size 1.

Surprising findings

• At 10% compression, ChunkKV on many-shot arithmetic reasoning matches or exceeds several token-level methods, and R1-Distill-Llama-8B degrades less than LLaMA-3.1-8B-Instruct under compression.
• World knowledge (MMLU) is nearly unaffected even at 10% compression, while GSM8K drops by over 35% on average.

Limitations

• Requires direct KV cache access, so limited to open-weight or self-hosted models.
• Attention-based shot importance score is a heuristic rather than a principled semantic importance measure.
• Effectiveness depends on prompts containing recoverable semantic units such as few-shot exemplars or sentences.

Experimental setup

Datasets: MMLU CommonsenseQA GSM8K HumanEval JailBreakV LongGenBench HotpotQA

Benchmarks: KVFundaBench MMLU CommonsenseQA GSM8K HumanEval JailBreakV LongGenBench HotpotQA

Metrics: accuracy pass@1 attack success rate relative performance change latency throughput

Baselines: StreamingLLM H2O SnapKV PyramidKV PyramidInfer ChunkKV Random Shot

Builds on: ChunkKV SCOPE H2O SnapKV StreamingLLM

Closest papers by meaning

• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models (NeurIPS 2025 · similarity 0.92)
• BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference (ICML 2026 · similarity 0.92)
• ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models (ICLR 2026 · similarity 0.91)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.90)
• RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.90)
• Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning (ICLR 2025 · similarity 0.90)