Research map / Large language models / Reasoning and chain of thought

KaVa: Latent Reasoning via Compressed KV-Cache Distillation

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes KaVa, which distills an R-KV-compressed teacher KV-cache (from a full CoT) into a PCCoT-style latent-reasoning student via KV matching plus CODI self-distillation, evaluated on GSM8k variants with Qwen2.5-0.5B and Llama3.2-1B/3B.

Problem

Latent reasoning lacks direct supervision for continuous thoughts, and performance degrades on long natural-language reasoning traces; compressed KV-caches lose token correspondence, so standard token/hidden-state distillation is ill-posed.

Approach

Compress the teacher's per-layer, per-head KV-cache with a redundancy- and importance-aware eviction (adapted R-KV) to the latent budget, then train the latent student's generated K and V directly to match it, combined with CODI self-distillation and a teacher CoT loss.

Key results

• On GSM8k-AUG-NL with Llama3.2-1B, KaVa reaches 55.7% on GSM8k vs. 50.1% for CODI and 51.1% for PCCoT, with 92% fewer forward passes than Full CoT at 3-6 per question range (62-92% fewer overall).
• With Llama3.2-3B on GSM8k-AUG-NL, KaVa gets 60.0% GSM8k vs. 55.9% for CODI, and KaVa's drop from AUG to AUG-NL is smaller than CODI and PCCoT's.
• On Qwen2.5-0.5B GSM8k-AUG-NL, KaVa reaches 44.4% vs. 20.2% for CODI and 19.1% for PCCoT, approaching Full CoT at 48.5%.

Surprising findings

• Decoded latent thoughts on GSM8k-AUG are often identical to the teacher's CoT tokens, while on GSM8k-AUG-NL they resist direct readout.
• CODI distillation loss caused training instabilities on MetaMathQA, so the authors set alpha_1=0 and used KV distillation alone.

Limitations

• Performance on MetaMathQA with full latent replacement is weak; hybrid reasoning with retained hard tokens was needed.
• Training is sensitive to data size; reduced datasets degrade KaVa accuracy and cannot be fully recovered by more epochs.
• Hyperparameters (loss type, alpha, normalization) are swept per model and dataset, and the method is limited to math word problems with small models.

Experimental setup

Datasets: GSM8k-AUG GSM8k-AUG-NL MetaMathQA GSM8k GSM8k-Hard SVAMP MATH500 MultiArith DeepMind-Mathematics

Benchmarks: GSM8k GSM8k-Hard SVAMP MATH500 MultiArith DeepMind-Mathematics

Metrics: test accuracy average forward passes

Baselines: Full CoT No-CoT CODI PCCoT iCoT Coconut

Builds on: CODI PCCoT Coconut R-KV iCoT

Closest papers by meaning

• Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression (NeurIPS 2026 · similarity 0.91)
• Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains (NeurIPS 2025 · similarity 0.91)
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models (NeurIPS 2025 · similarity 0.90)
• Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression (ICML 2026 · similarity 0.90)
• BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference (ICML 2026 · similarity 0.90)
• Efficient Reasoning via Constrained Optimization in Latent Space (NeurIPS 2026 · similarity 0.89)