Research map / Large language models / Reasoning and chain of thought

LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes why vanilla Soft Thinking underperforms token decoding, finds LLMs act as single-threaded (top-1-dominated) reasoners, and proposes Stochastic Soft Thinking with Gumbel-Softmax or Dirichlet sampling to break the greedy pitfall.

Problem

Soft Thinking (feeding probability-weighted soft tokens back into the LLM) is assumed to enable parallel exploration of reasoning paths, but it underperforms discrete token decoding on reasoning benchmarks and its actual mechanism is poorly understood.

Approach

Probing (JS divergence, logit lens, ROUGE-L against greedy traces) shows the model's prediction from a soft input is dominated by the top-1 token, creating a greedy feedback loop; injecting controlled randomness via Gumbel-Softmax (or Dirichlet) soft tokens breaks this loop and yields gains over vanilla Soft Thinking and, for Gumbel, over token sampling.

Key results

• Gumbel-Softmax stochastic soft thinking raises average accuracy over eight benchmarks to 79.55 (DeepSeek-R1-Distill-Qwen-32B), 83.04 (QwQ-32B), and 83.41 (Skywork-OR1-32B), versus vanilla soft thinking at 72.13, 80.06, and 79.21.
• On QwQ-32B and Skywork-OR1-32B, Gumbel soft thinking exceeds token sampling (82.35 and 82.99 average).
• Vanilla Soft Thinking averages 72.13 vs 78.50 for token sampling on DeepSeek-R1-Distill-Qwen-32B, and its ROUGE-L with the greedy trace is significantly higher than discrete token thinking.

Surprising findings

• Vanilla Soft Thinking performs worse than discrete token sampling and behaves like greedy decoding, with the soft input's prediction nearly identical to the top-1 token's prediction.
• Dirichlet sampling improves over vanilla soft thinking but does not beat token sampling, while Gumbel-Softmax does.

Limitations

• RL integration of stochastic soft thinking is left for future work, so RL gains are not demonstrated.
• Vanilla soft thinking and CoT comparisons rely on a single SGLang implementation and fixed hyperparameters (alpha=4.0, tau=0.5) chosen on the benchmarks.
• Analysis of single-threadedness is based mainly on QwQ-32B traces from AIME tasks.

Experimental setup

Datasets: AIME'24 AIME'25 MATH-500 AMC'23 GPQA-Diamond HumanEval MBPP LiveCodeBench

Benchmarks: AIME'24 AIME'25 MATH-500 AMC'23 GPQA-Diamond HumanEval MBPP LiveCodeBench

Metrics: accuracy Average@32 Pass@1 Pass@k ROUGE-L

Baselines: Token (Greedy) Token (Sampling) Soft (Vanilla) Soft (Dirichlet) COCONUT CoLaR

Builds on: Soft Thinking Chain-of-Thought COCONUT Gumbel-Max trick GRPO

Closest papers by meaning

• Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space (NeurIPS 2025 · similarity 0.94)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.89)
• Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning (NeurIPS 2025 · similarity 0.89)
• From Noise to Diversity: Random Embedding Injection in LLM Reasoning (NeurIPS 2026 · similarity 0.89)
• MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning (ICLR 2024 · similarity 0.89)
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.88)