Research map / Large language models / Reasoning and chain of thought

ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ThinKV, a thought-adaptive hybrid KV cache compression framework that uses attention sparsity to classify reasoning/execution/transition thoughts, quantizes tokens by thought importance, evicts tokens from less critical thoughts, and reuses evicted slots via a PagedAttention extension.

Problem

KV cache growth during long chain-of-thought generation in large reasoning models overwhelms GPU memory; existing token-level or recency-based compression degrades reasoning accuracy, and gather-based compaction after eviction adds large decode overhead.

Approach

Decompose CoT into reasoning, execution, and transition thoughts using layer-wise attention sparsity (KDE-calibrated thresholds), then assign precision by thought importance (R>E>T) and progressively evict tokens of earlier thought segments at transition thoughts, with a Continuous Thinking kernel that reuses evicted slots in place without compaction.

Key results

• ThinKV achieves near-lossless accuracy with under 5% of the original KV cache, and for R1-Llama-8B and AceReason-14B on AIME keeps under 4% accuracy drop using about 1.3% of the KV cache.
• On R1-Llama-8B (A100-80GB, 32K generation, 2048 budget), ThinKV raises max batch size from 13 to 290 and reaches 4688.4 vs 297.5 throughput over FullKV (about 15.8x).
• Throughput gains of up to 5.8x over R-KV (seq) and 3.6x over R-KV (ovl), with up to 3.2x higher iso-batch throughput than R-KV (seq) at batch 256 due to the CT kernel.

Surprising findings

• Quantization alone can inflate generation length by up to 5.1x on R1-Llama-8B, negating compression gains.
• Removing transition (backtracking) thoughts can cause the model to loop endlessly, so high-importance T outliers must be retained.

Limitations

• Proof-of-concept integration in Hugging Face Transformers with a Triton PagedAttention rather than an optimized serving engine.
• Thought types are assigned via sparsity thresholds and keyword-based interpretation is only illustrative, so semantic correspondence is approximate.

Experimental setup

Datasets: s1K (100 calibration prompts) MATH-500 AIME GSM8K LiveCodeBench

Benchmarks: MATH-500 AIME GSM8K LiveCodeBench

Metrics: pass@1 throughput time per output token latency batch size

Baselines: H2O RaaS R-KV LazyEviction KIVI PM-KVQ

Builds on: PagedAttention KIVI R-KV H2O NVFP4

Closest papers by meaning

• BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference (ICML 2026 · similarity 0.92)
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models (NeurIPS 2025 · similarity 0.92)
• Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression (ICML 2026 · similarity 0.91)
• Value-Aware Stochastic KV Cache Eviction for Reasoning Models (NeurIPS 2026 · similarity 0.91)
• Training Large Reasoning Models Efficiently via Progressive Thought Encoding (ICLR 2026 · similarity 0.90)
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.90)