Research map / Large language models / Reasoning and chain of thought

Geometric Latent Reasoning Induces Shorter Generations in LLMs

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Geometric Latent Reasoning (GLR), which trains a linear transition head to predict embedding-space displacements anchored to CoT traces, so that K initial latent steps replace explicit reasoning and yield shorter generations on math benchmarks with Qwen3 models.

Problem

Explicit chain-of-thought reasoning is expensive and length-sensitive, while existing latent reasoning methods use unconstrained hidden states or external modules that misalign with the pretrained embedding geometry.

Approach

Model reasoning as a path in the pretrained token-embedding space and train a lightweight linear head to predict per-step embedding displacements (position-discounted MSE against CoT embedding differences); at inference, K continuous latent steps replace an initial explicit reasoning prefix.

Key results

• On MATH500 with Qwen3-1.7B at a 512-step budget, CoT-SFT solves nearly 0% while GLR K=10 solves over 40%.
• On MATH500 with Qwen3-1.7B, median correct CoT-SFT generation is about 2,000 tokens versus roughly 350 total steps for GLR K=10 and K=20.
• On SVAMP, CoT-SFT correct solutions have median lengths of roughly 500-700 tokens, while GLR solves the same problems in roughly 100 total steps.

Surprising findings

• Shorter generations emerge without any explicit length objective.
• At large budgets (2048/4096 steps), CoT-SFT recovers and often outperforms GLR in final accuracy.

Limitations

• Training limited to 10K CoT examples and small Qwen3 models (0.6B and 1.7B), with accuracy drift at large latent budgets (K=80, 100).
• Evaluated only on mathematical reasoning.
• No wall-clock latency comparison because GLR uses a custom HuggingFace decoder while CoT-SFT uses vLLM.

Experimental setup

Datasets: Open-R1 Mixture-of-Thoughts (math split, 10K sample) GSM8K SVAMP MultiArith MATH500 AMC23 OlympiadBench

Benchmarks: GSM8K MATH500 SVAMP MultiArith AMC23 OlympiadBench

Metrics: pass@1 accuracy (greedy) generation length (steps) among correct solutions accuracy under generation budget

Baselines: CoT-SFT

Builds on: chain-of-thought soft tokens Coconut

Closest papers by meaning

• Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression (NeurIPS 2026 · similarity 0.92)
• Efficient Reasoning via Constrained Optimization in Latent Space (NeurIPS 2026 · similarity 0.91)
• Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.90)
• Tracing the Traces: Latent Temporal Signals for Efficient and Accurate Reasoning (ICLR 2026 · similarity 0.90)
• LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning (ICLR 2026 · similarity 0.90)
• AlgoTrace: Algorithmic Primitives and Compositional Geometry of Reasoning in Language Models (ICML 2026 · similarity 0.89)