Research map / Large language models / Reasoning and chain of thought

Soft Tokens, Hard Truths

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RL fine-tuning of continuous (soft/fuzzy) chain-of-thought tokens by injecting Gaussian noise into soft-token embeddings, enabling RLOO without ground-truth CoTs, and finds parity at pass@1 and gains at pass@32 on math tasks.

Problem

Continuous chain-of-thought reasoning is hard to train: backpropagating through continuous steps is costly and prior methods need ground-truth discrete CoTs or are inference-only, so it is unclear whether soft thinking helps trained models.

Approach

Make soft-thinking CoTs trainable by RL: feed the probability-weighted mixture of embeddings plus Gaussian noise on the input embeddings, which supplies exploration and a tractable Gaussian log-likelihood for REINFORCE-style updates (RLOO), with no ground-truth CoT supervision.

Key results

• Soft and fuzzy RLOO training reach greedy pass@1 comparable to hard-token training across models and datasets (e.g., Llama-3B on GSM8K: 76.7 fuzzy, 77.2 soft vs 75.9 hard).
• Soft/fuzzy training improves sample pass@32 on several settings, e.g., Llama-3B GSM8K soft 97.9 vs hard 94.1, and OlympiadBench 58.6 vs 49.3-ish range for Llama-3B.
• On Llama-8B trained on GSM8K and tested on MATH-500, hard training drops greedy to 20.2% while fuzzy and soft recover to about 44.6-44.7%.

Surprising findings

• Soft inference on hard-trained models does not improve results, contrary to previously reported gains from soft thinking.
• Hard (discrete) inference performs best for all training methods, even for models trained with soft or fuzzy CoTs.

Limitations

• Pass@1 is only at parity, not improved, relative to discrete training.
• Greedy-soft and soft-inference gains reported by prior work were not reproduced; benefits are mainly at pass@32 and out-of-domain NLL.
• Evaluation limited to small instruct models and math domains.

Experimental setup

Datasets: GSM8K MATH DeepScaleR HellaSwag MMLU ARC

Benchmarks: GSM8K MATH-500 OlympiadBench HellaSwag ARC MMLU

Metrics: greedy pass@1 sample pass@1 sample pass@32 negative log-likelihood of correct answer

Baselines: Soft Thinking Coconut CoDi hard-token RLOO training no finetune base model

Builds on: Soft Thinking Coconut RLOO GRPO

Closest papers by meaning

• Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.91)
• Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space (NeurIPS 2025 · similarity 0.89)
• SIM-CoT: Supervised Implicit Chain-of-Thought (ICLR 2026 · similarity 0.89)
• Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning (NeurIPS 2025 · similarity 0.89)
• CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs (ICLR 2026 · similarity 0.89)
• Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026 · similarity 0.89)