Research map / Large language models / Reasoning and chain of thought

Beyond Speedup - Utilizing KV Cache for Sampling and Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Treats the transformer KV cache as a free, reusable representation and shows it supports Chain-of-Embedding self-evaluation (KV-CoE) and adaptive fast/slow thinking switching (KVClassifier) without storing hidden states.

Problem

KV caches are used only to speed up decoding, while downstream uses of internal representations (self-evaluation, adaptive reasoning) require storing full hidden states at extra memory and compute cost. The paper asks whether the KV cache can serve these downstream roles at near-zero cost.

Approach

Pool the already-computed KV cache (token-wise for KV-CoE, mean-pooled over tokens/layers for KVClassifier) to form trajectory features for CoE confidence scoring and difficulty scores that gate fast vs. slow reasoning via control tokens, avoiding hidden-state storage.

Key results

• On MATH with Qwen2-7B-Instruct, KV-CoE-C reaches 84.12 AUROC and 44.82 FPR95 versus 76.68 AUROC for CoE-C.
• On MATH500 with Qwen3-8B, KV-Generative reduces average tokens from 4,150 to 727 (5.7x fewer) at accuracy 0.578 vs 0.610 for full reasoning.
• On GSM8K with Qwen3-8B, KV-Classification reaches 0.914 accuracy vs 0.933 for full reasoning while using 66.1% fewer tokens.

Surprising findings

• KV-derived embeddings are far weaker than a dedicated embedding model on MTEB classification (e.g., 0.353 vs 0.882 on AmazonCounterfactual), yet still suffice for the CoE and switching tasks.
• Layer-wise KV-CoE aggregation performs substantially worse than token-wise aggregation, so KV cache should not be treated as a drop-in replacement for layer-wise hidden states.

Limitations

• KV caches are weak general-purpose embeddings and unsuitable for global semantic retrieval.
• Layer-wise KV aggregation performs poorly, limiting the method to token-centric pooling.
• Evaluation limited to a few models and math/QA reasoning datasets; efficiency overhead not quantified.

Experimental setup

Datasets: GSM8K MATH MATH500 TheoremQA MTEB classification subsets

Benchmarks: MATH TheoremQA GSM8K MATH500 MTEB classification (AmazonCounterfactual, DBpedia, FinancialPhrasebank, TweetTopic)

Metrics: AUROC FPR95 accuracy average token count

Baselines: MaxProb PPL Entropy CoE-R CoE-C Fast Thinking Full Reasoning gemini-embedding-001

Builds on: Chain-of-Embedding KV cache vLLM MTEB

Closest papers by meaning

• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models (NeurIPS 2025 · similarity 0.90)
• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.90)
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction (NeurIPS 2025 · similarity 0.89)
• RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse (ICML 2026 · similarity 0.89)
• KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025 · similarity 0.89)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.89)