Research map / Large language models / Reasoning and chain of thought

Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLTT, a GRPO alternative that applies the policy gradient to next-token distributions from every loop of a looped language model (Ouro), yielding gains over GRPO on math and non-math reasoning benchmarks.

Problem

Standard RL objectives like GRPO reward only the final loop's latent state in looped language models (LoopLMs), mismatching the multi-step latent computation and causing prior RLVR attempts to fail.

Approach

RLTT replaces the terminal-loop log-probability in the policy gradient with a weighted sum of per-loop log-probabilities (weights from exit probabilities, progressive, or uniform schemes), giving dense trajectory-level credit without external verifiers.

Key results

• On the 2.6B scale, RLTT reaches 51.2% average on math benchmarks vs 42.5% for GRPO (+10.9 points average gain over GRPO on math benchmarks), and 86.0% on MATH-500 vs 79.0%.
• On the 1.4B scale, RLTT improves mean math accuracy from 41.7% (GRPO) to 46.0%, with AIME24 at 26.7% vs 16.7%.
• RLTT transfers zero-shot to non-math benchmarks, with GPQA going from 19.7% to 38.4% at the 2.6B scale; average non-math accuracy 71.8% vs 65.2%.

Surprising findings

• RLTT-trained policies converge to shorter responses than GRPO despite no explicit brevity reward.
• Training with RLTT on MATH transfers to non-math benchmarks such as GPQA, nearly doubling GRPO performance.

Limitations

• Per-loop log-probability retention increases memory, halving per-GPU token packing.
• Fixed loop depth during training and inference; forgoes Ouro's adaptive early exit.
• Method is specific to looped architectures and not applicable to standard transformers.

Experimental setup

Datasets: MATH

Benchmarks: MATH-500 AIME24 AIME26 BeyondAIME GSM8K ARC-C MMLU-ST GPQA MBPP

Metrics: accuracy (Pass@1) pass@k GSNR paired t-test significance

Baselines: GRPO SFT Qwen3-1.7B+GRPO Qwen3-4B+GRPO DeepSeek-R1-1.5B DeepSeek-R1-7B

Builds on: GRPO Ouro VERL

Closest papers by meaning

• LoopRPT: Reinforcement Pre-Training for Looped Language Models (NeurIPS 2026 · similarity 0.92)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.92)
• RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.91)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.91)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.91)
• Hybrid Latent Reasoning via Reinforcement Learning (NeurIPS 2025 · similarity 0.91)