Research map / Large language models / Reasoning and chain of thought
Hybrid Latent Reasoning via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HRPO, an on-policy RL method that trains LLMs to do hybrid latent reasoning by gating sampled token embeddings with projected hidden states, without CoT traces.
Problem
Latent reasoning methods are incompatible with the discrete nature of LLM autoregressive generation and typically require CoT traces and costly multi-stage training.
Approach
A learnable gating mechanism mixes sampled token embeddings with probability-weighted embedding interpolations of the previous hidden state during reasoning; initialization favors tokens, and RL (GRPO-style, outcome reward) gradually shifts weight toward latent features. Sampling keeps stochasticity so RL works without CoT data.
Key results
- HRPO with Qwen2.5-3B reaches 0.380 average EM on knowledge QA, outperforming the strongest 7B RAG baseline (0.335 average) and GRPO (0.367).
- HRPO with Qwen2.5-3B averages 0.700 on STEM benchmarks, matching the best 7B baseline (MAmmoTH2-8B, 0.652 average) and beating GRPO (0.691).
- On GSM8k and MATH with Qwen2.5-1.5B, HRPO scores 0.720 and 0.518, above Coconut (0.315 GSM8k), CODI (0.658 GSM8k, 0.419 MATH) and distilled CoT (0.706, 0.503).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with HRPO instead of PPO | Qwen2.5-1.5B-Instruct · Knowledge average | exact match: 0.307 -> 0.337 | Helped |
| train with HRPO instead of GRPO | Qwen2.5-3B-Instruct · Knowledge average | exact match: 0.367 -> 0.380 | Helped |
| train with HRPO instead of GRPO | Qwen2.5-1.5B-Instruct · STEM average | accuracy: 0.607 -> 0.617 | Helped |
| train with HRPO instead of GRPO | Qwen2.5-3B-Instruct · STEM average | accuracy: 0.691 -> 0.700 | Helped |
| use final-layer hidden states directly instead of hybrid gated representations (ablation) | HRPO on Qwen2.5-1.5B · MATH | reward | Hurt |
| use interpolated embeddings instead of hybrid gated representations (ablation) | HRPO on Qwen2.5-1.5B · MATH | reward | Hurt |
| set the gate initialization range to [0.95 - 0.999] instead of [0.98 - 0.999] (ablation) | HRPO on Qwen2.5-1.5B · Knowledge average | exact match: 0.329 -> 0.328 | No clear effect |
| set the gate initialization range to [0.99 - 0.999] instead of [0.98 - 0.999] (ablation) | HRPO on Qwen2.5-1.5B · STEM average | accuracy: 0.608 -> 0.614 | Helped |
| vary gate initialization across [0.95 - 0.999], [0.98 - 0.999], and [0.99 - 0.999] (ablation) | HRPO on Qwen2.5-3B · Knowledge and STEM averages | accuracy: Knowledge: 0.695, 0.691, 0.696; STEM: 0.377, 0.374, 0.375 | Mixed |
| vary embedding temperature τ across 0.3, 0.5, 0.7, and 0.9 (ablation) | HRPO on Qwen2.5-1.5B and Qwen2.5-3B · STEM and knowledge benchmarks | accuracy: GSM8k: 0.717, 0.720, 0.705, 0.694; MATH: 0.518, 0.516, 0.507, 0.514 | Mixed |
| increase the initial hidden ratio by lowering r_min (ablation) | HRPO · training convergence and completion length | completion length | Helped |
Surprising findings
- Feeding raw final-layer hidden states as next inputs degrades generation to nonsensical rollouts with zero reward, while the interpolated-embedding variant collapses after a few hundred steps.
- Initializing with a smaller r_min (larger initial hidden ratio) shortens completions on some runs, and HRPO-trained models show cross-lingual and rare-token reasoning patterns.
Limitations
- Introduces additional computation overhead during rollouts.
- On-policy design may reduce large-scale training efficiency; each trajectory used for one update.
- Continuous representations are less transparent and interpretable; errors include cross-lingual mistakes and format violations.
Experimental setup
Datasets: Natural Questions HotpotQA GSM8k MATH MMLU auxiliary train ARC-C train English Wikipedia 2020 dump
Benchmarks: NQ TriviaQA HotpotQA 2WikiMultiHopQA Bamboogle GSM8k MATH MATH500 MMLU-STEM ARC-Challenge
Metrics: exact match accuracy average accuracy paired t-test significance
Baselines: SFT RAG PPO GRPO IRCoT Search-o1 Coconut CODI Distilled CoT DeepSeekMath-7B Gemma-2-9B Qwen2.5-7B
Builds on: GRPO Coconut CODI PPO gated recurrence models
Closest papers by meaning
- Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.91)
- Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space (NeurIPS 2026 · similarity 0.91)
- Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models (ICML 2026 · similarity 0.91)
- Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search (ICML 2025 · similarity 0.90)
- Native Reasoning Models: Training Language Models to Reason on Unverifiable Data (ICLR 2026 · similarity 0.90)
- Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.