Research map / Large language models / Reasoning and chain of thought

Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokens

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LT-Tuning, a three-stage post-training framework that builds latent thought tokens by fusing hidden states with vocabulary-embedding predictions and switches dynamically between latent and explicit reasoning.

Problem

Latent reasoning methods that recycle raw hidden states as input embeddings suffer distribution mismatch and feature collapse, especially in models with untied embeddings, while assistant-model approaches suffer misalignment; static latent allocation is also inefficient.

Approach

Construct each latent token as a convex fusion of the intermediate hidden state (context) and the probability-weighted vocabulary embedding (prediction), trained via a curriculum from explicit CoT to confidence-triggered latent tokens.

Key results

• LT-Tuning averages 36.4% (1B), 52.4% (3B), and 68.8% (8B) across four math benchmarks, best among compared latent baselines at each scale.
• On Llama-3.1-8B, Coconut drops to 41.5% average versus 61.7% for explicit CoT, while LT-Tuning reaches 68.8%; adding an adapter gives 70.3%.
• Removing the fusion stage (w/o Stage 3) reduces 8B average accuracy by 23.5 points, versus 8.0 points at 3B.

Surprising findings

• Coconut falls below explicit CoT on Llama-3.1-8B (41.5% vs 61.7%) despite being competitive at smaller scales.
• For the 8B model, removing latent reasoning entirely (w/o Latent, 61.6%) outperforms removing fusion (w/o Stage 3, 45.3%), suggesting poorly built latent tokens can hurt more than none.

Limitations

• Evaluation limited to mathematical word problems with small GSM8K-based training, plus one GPQA-Diamond transfer experiment.
• The fusion weight alpha is fixed (0.6) rather than learned or tuned per model, and the theoretical justification does not claim global optimality.
• Latent token budgets and confidence thresholds are model-specific hyperparameters that complicate cross-scale comparison.

Experimental setup

Datasets: GSM8K Bespoke-Stratos-17k

Benchmarks: GSM8K-NL ASDiv-Aug MultiArith SVAMP GPQA-Diamond

Metrics: accuracy

Baselines: Explicit CoT Coconut Soft-Thinking SoftCoT SemCoT

Builds on: Coconut Soft-Thinking SoftCoT chain-of-thought

Closest papers by meaning

• Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in its Latent Thoughts (ICLR 2026 · similarity 0.90)
• Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.90)
• Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression (NeurIPS 2026 · similarity 0.90)
• Think in Latent, Explain in Language: Self-Explainable Latent Reasoning (ICML 2026 · similarity 0.89)
• Efficient Post-Training Refinement of Latent Reasoning in Large Language Models (AAAI 2026 · similarity 0.89)
• PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space (ICML 2026 · similarity 0.89)