Research map / Large language models / Agents, code and math
Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Soft Reasoning, which perturbs the first-token embedding with Gaussian noise and uses Bayesian optimisation with a self-verifier reward to search for better greedy reasoning paths.
Problem
LLMs struggle with complex reasoning because temperature-based sampling gives limited, noisy diversity and prompt-driven search (CoT, self-consistency, tree search) is inefficient.
Approach
Inject Gaussian perturbations into the first answer token's embedding (mapped to a special token), decode greedily so each embedding maps to one answer, and optimise the embedding with expected-improvement Bayesian optimisation over a random low-dimensional projection, using a same-model verifier plus coherence log-probability as the reward.
Key results
- On LLaMA3-8B-Ins few-shot GSM8K, Soft Reasoning reaches 84.3% vs 80.7% for RAP and 80.6% for the best SC setting.
- Soft Reasoning uses about 6.19% of RAP's input tokens and 14.3% of its inference time on the reported benchmarks.
- Zero-shot GSM8K coverage with LLaMA3-8B-Ins reaches 91.8%, compared with 84.5% for FIRE and 85.3% for CoT-Decoding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add verifier score to the objective (ablation) | Soft Reasoning · GSM8K zero-shot, LLaMA3-8B-Ins | accuracy (%): 76.8$_{\pm1.0}$ -> 79.4$_{\pm1.2}$ | Helped |
| Add coherence term to the objective (ablation) | Soft Reasoning · GSM8K zero-shot, LLaMA3-8B-Ins | accuracy (%): 77.4$_{\pm2.1}$ -> 79.4$_{\pm1.2}$ | Helped |
| Use Expected Improvement instead of PI acquisition (ablation) | Soft Reasoning · GSM8K zero-shot, LLaMA3-8B-Ins | accuracy (%): 74.6±1.5 -> 79.4±1.2 | Helped |
| Optimise only the first token embedding rather than 2 initial token embeddings (ablation) | Soft Reasoning · GSM8K zero-shot, LLaMA3-8B-Ins | accuracy (%): 75.0$_{\pm0.8}$ -> 79.4$_{\pm1.2}$ | Helped |
| Optimise only the first token embedding rather than 5 initial token embeddings (ablation) | Soft Reasoning · GSM8K zero-shot, LLaMA3-8B-Ins | accuracy (%): 69.7$_{\pm3.5}$ -> 79.4$_{\pm1.2}$ | Helped |
| Place perturbed special token last in the prompt instead of first (ablation) | Soft Reasoning · GSM8K zero-shot, LLaMA3-8B-Ins | accuracy (%): 77.7$_{\pm2.5}$ -> 79.4$_{\pm1.2}$ | Helped |
| Place perturbed special token last in the prompt instead of middle (ablation) | Soft Reasoning · GSM8K zero-shot, LLaMA3-8B-Ins | accuracy (%): 78.5$_{\pm3.1}$ -> 79.4$_{\pm1.2}$ | Helped |
| Use Soft Reasoning instead of RAP | LLaMA3-8B-Ins · GSM8K, GSM-Hard, SVAMP, and StrategyQA 8-shot | accuracy (%): 80.7 -> 84.3; 32.7 -> 35.7; 87.9 -> 90.2; 73.4 -> 75.6 | Helped |
Surprising findings
- Optimising the embedding of more than one initial token degrades accuracy, with GSM8K dropping from 79.4% at k=1 to 52.2% at k=20 in zero-shot.
- Multi-candidate generation verifiers outperform single-candidate judging, with Single-Judge verifier accuracy on GSM-Hard at 60.9% versus 78.2% for Multi-Generate.
Limitations
- Relies on a same-model verifier that may give unreliable feedback.
- Operates only on the first token, so interpretability of perturbation effects is limited.
- Reported gains are small on some tasks, and the Qwen2-7B few-shot StrategyQA result trails RAP.
Experimental setup
Datasets: GSM8K GSM-Hard SVAMP StrategyQA
Benchmarks: GSM8K GSM-Hard SVAMP StrategyQA AIME-2024
Metrics: accuracy coverage input/output token count inference time
Baselines: CoT Self-Consistency FIRE CoT-Decoding RAP Trainable Prefix Scorer Constrained Fine-tuning
Builds on: chain-of-thought self-consistency CoT-Decoding RAP random embeddings Bayesian optimisation
Closest papers by meaning
- Think Then Embed: Generative Context Improves Multimodal Embedding (ICLR 2026 · similarity 0.88)
- Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space (NeurIPS 2025 · similarity 0.88)
- From Noise to Diversity: Random Embedding Injection in LLM Reasoning (NeurIPS 2026 · similarity 0.88)
- T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.88)
- TSLM: Tree-Structured Language Modeling for Divergent Thinking (ICLR 2026 · similarity 0.88)
- AlphaZero-Like Tree-Search can Guide Large Language Model Decoding and Training (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.