Research map / Large language models / Reasoning and chain of thought
Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Soft Thinking, a training-free decoding method that feeds probability-weighted mixtures of token embeddings (concept tokens) back into the LLM during reasoning, with an entropy-based Cold Stop, improving accuracy and reducing tokens on math and coding benchmarks.
Problem
Standard chain-of-thought commits to one sampled discrete token per step, restricting expressiveness and exploration of reasoning paths; latent/continuous reasoning methods like COCONUT suffer from hidden-state vs. input-embedding mismatch in large decoupled-embedding models.
Approach
During intermediate thinking, feed back the next-token distribution as a concept token, i.e., the embedding is the probability-weighted sum of vocabulary embeddings (restricted to top-n tokens), with no training; an entropy-based Cold Stop inserts end-of-thinking when the model is confident for several consecutive steps.
Key results
- Soft Thinking improves average math Pass@1 for QwQ-32B from 83.84 to 86.32 (+2.48) and AIME 2024 by 6.45 points.
- On DeepSeek-R1-Distill-Qwen-32B math, generation length drops 22.4% (4995 to 3875 tokens) while accuracy rises 1.71 points.
- Training-free COCONUT produces no correct answers on AIME 2024 and LiveCodeBench, while Soft Thinking reaches 83.33 on AIME 2024 with QwQ-32B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace discrete CoT tokens with probability-weighted concept tokens and use Cold Stop | QwQ-32B · Math benchmarks (Avg.) | Pass@1: 83.84 -> 86.32 | Helped |
| Replace discrete CoT tokens with probability-weighted concept tokens and use Cold Stop | QwQ-32B · Coding benchmarks (Avg.) | Pass@1: 85.70 -> 86.18 | Helped |
| Replace discrete CoT tokens with probability-weighted concept tokens and use Cold Stop | DeepSeek-R1-Distill-Qwen-32B · Math benchmarks (Avg.) | Pass@1: 81.32 -> 83.03 | Helped |
| Replace discrete CoT tokens with probability-weighted concept tokens and use Cold Stop | DeepSeek-R1-Distill-Qwen-32B · Coding benchmarks (Avg.) | Pass@1: 83.23 -> 84.13 | Helped |
| Replace discrete CoT tokens with probability-weighted concept tokens and use Cold Stop | DeepSeek-R1-Distill-Llama-70B · Math benchmarks (Avg.) | Pass@1: 81.31 -> 82.42 | Helped |
| Replace discrete CoT tokens with probability-weighted concept tokens and use Cold Stop | DeepSeek-R1-Distill-Llama-70B · Coding benchmarks (Avg.) | Pass@1: 83.14 -> 83.84 | Helped |
| Add Cold Stop to Soft Thinking (ablation) | Soft Thinking with QwQ-32B · AIME 2024 | accuracy: 73.33 -> 83.33 | Helped |
| Add Cold Stop to Soft Thinking (ablation) | Soft Thinking with QwQ-32B · LiveCodeBench | accuracy: 56.98 -> 62.72 | Helped |
| Add Cold Stop to Soft Thinking (ablation) | Soft Thinking with QwQ-32B · AIME 2024 | Generation length (all): 12,991 -> 11,445 | Helped |
| Add Cold Stop to Soft Thinking (ablation) | Soft Thinking with QwQ-32B · LiveCodeBench | Generation length (all): 13,705 -> 12,537 | Helped |
| Use the mean of top-n embeddings instead of probability-weighted embeddings (ablation) | Soft Thinking with QwQ-32B · AIME 2024 | accuracy: 83.33 -> 6.66 | Hurt |
| Use the mean of top-n embeddings instead of probability-weighted embeddings (ablation) | Soft Thinking with QwQ-32B · LiveCodeBench | accuracy: 62.72 -> 7.49 | Hurt |
Surprising findings
- Training-free COCONUT fails entirely (0 accuracy) while the simple averaged-embedding variant gets only a few correct answers, showing naive continuous feedback is OOD-fragile.
- Without Cold Stop, generation collapses into repetition until maximum length, indicating OOD instability of concept tokens.
Limitations
- Concept tokens are out-of-distribution for models never trained on them, risking instability or collapse on long chains.
- Cold Stop is a heuristic that does not fundamentally resolve OOD issues.
- Evaluated only on three open reasoning models (32B and 70B) and selected hyperparameters on test results.
Experimental setup
Datasets: MATH500 AIME 2024 GSM8K GPQA-Diamond HumanEval MBPP LiveCodeBench
Benchmarks: MATH500 AIME 2024 GSM8K GPQA-Diamond HumanEval MBPP LiveCodeBench
Metrics: Pass@1 generation length (tokens of correct solutions)
Baselines: CoT Thinking CoT Thinking (Greedy) COCONUT-TF Average Embedding
Builds on: chain-of-thought COCONUT SGLang
Closest papers by meaning
- LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking (ICLR 2026 · similarity 0.94)
- Soft Tokens, Hard Truths (ICLR 2026 · similarity 0.89)
- MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning (ICLR 2024 · similarity 0.89)
- Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration (ICML 2025 · similarity 0.88)
- Learning to Reason via Mixture-of-Thought for Logical Reasoning (ICLR 2026 · similarity 0.88)
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.