Research map / Large language models / Reasoning and chain of thought
Learning to Reason over Continuous Tokens with Reinforcement Learning
Yiran Zhao, Yuhui Xu, Doyen Sahoo, Caiming Xiong, Junnan Li
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learning to reason over continuous tokens in embedding space with reinforcement learning, aiming to keep the efficiency of latent reasoning while retaining clarity of discrete CoT.
Problem
Discrete chain-of-thought reasoning is computationally and memory expensive due to verbose intermediate steps, while existing latent reasoning in embedding space is efficient but often loses clarity and interpretability.
Approach
Performs reasoning over continuous token embeddings and trains the model with reinforcement learning so that latent reasoning steps are optimized directly for task success rather than imitating discrete CoT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GRPO reinforcement learning stage after cold start (ablation) | HyRea (Qwen2.5-7B-Instruct) · MATH-500 | pass@1 accuracy: 71.8 -> 83.6 | Improved |
| add RL stage (ablation) | HyRea (Qwen2.5-32B-Instruct) · MATH-500 | pass@1 accuracy: 79.8 -> 84.4 | Improved |
| use hybrid latent/explicit reasoning instead of SFT+RL | Qwen2.5-7B-Instruct · MATH-500 | pass@1 accuracy and tokens: 84.2 -> 83.6 accuracy; 698 -> 387 tokens | Mixed |
| use hybrid reasoning instead of pure latent reasoning (Coconut) | Qwen2.5-7B-Instruct · MATH-500 | pass@1 accuracy: 70.4 -> 83.6 | Improved |
| select latent-replaced steps by low entropy instead of randomly (ablation) | HyRea · MATH-500 tokens | avg output tokens: 309 -> 287 tokens (accuracy 83.4 -> 83.6) | Improved |
| select latent-replaced steps by low entropy instead of randomly (ablation) | HyRea · AMC23 | pass@1 accuracy: 49.4 -> 48.2 | Hurt |
| increase number of latent tokens c from 1 to 8 (ablation) | HyRea · math accuracy | accuracy: over 80% -> below 10% | Hurt |
| apply HyRea-trained model to general tasks versus SFT+RL | Qwen2.5-7B-Instruct · MMLU | accuracy: 73.4 -> 68.6 (tokens 102 -> 53) | Hurt |
| apply HyRea-trained model to GPQA versus SFT+RL | Qwen2.5-7B-Instruct · GPQA | accuracy: 29.8 -> 27.4 (tokens 1083 -> 685) | Mixed |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- Soft Thinking — Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space (NeurIPS 2025) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Experimental setup
Builds on: chain-of-thought prompting
Closest papers by meaning
- Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.91)
- Efficient Post-Training Refinement of Latent Reasoning in Large Language Models (AAAI 2026 · similarity 0.91)
- Hybrid Latent Reasoning via Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space (NeurIPS 2026 · similarity 0.91)
- LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts (ICLR 2026 · similarity 0.90)
- $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
