Research map / Large language models / Reasoning and chain of thought
Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoLaR, which compresses multiple reasoning token embeddings into dense latents predicted by a probabilistic latent head, trained with SFT (next-compressed-embedding prediction) and then GRPO, to shorten LLM reasoning chains.
Problem
Token-level chain-of-thought reasoning is computationally expensive, and existing latent reasoning methods (Coconut, CODI) use fixed-length, deterministic latent chains that limit efficiency and exploration.
Approach
Train an LLM to predict the next compressed embedding (sum of c consecutive reasoning token embeddings scaled by 1/sqrt(c)) with a probabilistic latent head sampled at a randomly chosen compression factor, so the test-time compression factor can be set by prompting; then apply GRPO over the stochastic latent paths to explore and exploit shorter chains.
Key results
- CoLaR-5 achieves 14.1% higher average accuracy than Coconut on four GSM datasets at comparable compression (41.7% vs 27.6%, chain length 4.57 vs 6.00).
- CoLaR-2 reaches 48.8% average accuracy vs 53.6% for CoT (a 4.8% drop) with about 53% shorter reasoning chains.
- RL on MATH raises DeepSeek-R1-Distill-Qwen-1.5B CoLaR accuracy from 8.94% to 14.3% (+5.36%) while cutting chain length by 82.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoLaR-5 instead of Coconut | CoLaR · four grade-school math datasets (average) | accuracy: +14.1% accuracy | Helped |
| Use CoLaR-2 instead of explicit CoT | CoLaR · four grade-school math datasets (average) | accuracy: 4.8% decrease | Hurt |
| Use CoLaR-2 instead of explicit CoT | CoLaR · four grade-school math datasets (average) | reasoning chain length: 53% reduction | Helped |
| Add GRPO post-training to CoLaR-NLL | CoLaR-NLL on MATH · MATH | accuracy: 8.94% -> 14.3% (+5.36%↑) | Helped |
| Add GRPO post-training to CoLaR-NLL | CoLaR-NLL on MATH · MATH | reasoning chain length: 56.8 -> 9.79 (82.8%↓) | Helped |
| Remove per-token reward/loss averaging (ablation) | CoLaR-NLL-RL · MATH, DeepSeek-R1-Distill-Qwen-1.5B | reasoning chain length: 9.79 -> 128 | Hurt |
| Remove per-token reward/loss averaging (ablation) | CoLaR-NLL-RL · MATH, Llama-3.2-1B-Instruct | accuracy: 7.08 -> 0.00 | Hurt |
| Use a deterministic latent head (ablation) | CoLaR · GSM8k-Aug | accuracy: 26.8 -> 26.7 | No clear effect |
| Remove compressed reasoning tokens from language-model supervision (ablation) | CoLaR · GSM8k-Aug | accuracy: 26.8 -> 24.8 | Hurt |
| Remove compressed reasoning tokens from language-model supervision (ablation) | CoLaR · GSM8k-Aug | accuracy: 40.1 -> 39.1 | Hurt |
| Use mean pooling to compress embeddings (ablation) | CoLaR · GSM8k-Aug | accuracy: 26.8 -> 20.6 | Hurt |
| Train with NLL latent loss instead of soft-MSE (ablation) | CoLaR · GSM8k-Aug | accuracy: 26.8 -> 20.3 | Hurt |
Surprising findings
- Training with the NLL latent loss under-fits simpler datasets, so the authors introduce a soft-MSE loss with entropy regularization to fit them better.
- Averaging the GRPO loss per token, rather than normalizing by a constant, keeps reasoning length from collapsing to the maximum limit and drives shorter chains.
Limitations
- On most benchmarks CoLaR only approximates explicit CoT and does not surpass it (GPQA is the exception).
- Cannot generalize to non-integer compression factors or to factors above the maximum training factor r_max.
- Experiments are limited to grade-school and competition math tasks.
Experimental setup
Datasets: GSM8k-Aug MATH
Benchmarks: GSM8k-Aug GSM-Hard SVAMP MultiArith MATH MATH-500 GPQA
Metrics: accuracy reasoning chain length
Baselines: CoT iCoT Coconut CODI (reproduced as Distill) TokenSkip
Builds on: Coconut CODI GRPO DAPO
Closest papers by meaning
- Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression (ICML 2026 · similarity 0.93)
- Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026 · similarity 0.92)
- Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression (NeurIPS 2026 · similarity 0.91)
- When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Models (ICLR 2026 · similarity 0.91)
- KaVa: Latent Reasoning via Compressed KV-Cache Distillation (ICLR 2026 · similarity 0.91)
- Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoning (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.