Research map / Large language models / Reasoning and chain of thought

LoopRPT: Reinforcement Pre-Training for Looped Language Models

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LoopRPT, a reinforcement pre-training framework that assigns step-wise RL rewards to latent iterations of looped language models (Ouro), using an EMA teacher, entropy-based hard-token selection, and noisy latent rollouts.

Problem

Existing RL paradigms (RLVR) target output tokens, which mismatches looped language models whose reasoning unfolds implicitly in latent iterations; sparse token-level rewards give poor credit assignment for adaptive exit and latent computation.

Approach

Reframes next-token prediction as a next-token reasoning task and assigns dense step-wise rewards to latent loop steps (accuracy gain over an EMA teacher baseline minus a difficulty-aware time penalty), training on high-entropy hard tokens with Gaussian-perturbed latent rollouts and GRPO-style group normalization for the exit policy.

Key results

• On Ouro-2.6B, LoopRPT raises Hard-split Adaptive accuracy from 34.35 to 37.24 while cutting average steps from 3.51 to 2.28 on next-token reasoning.
• On Ouro-2.6B, GSM8K rises from 81.76 to 85.36 and MBPP+ improves by about 0.27 over Ouro (65.08 vs 64.81), with HumanEval+ up 1.82.
• On Ouro-1.4B, MBPP improves by +2.38 and MBPP+ by +2.91 over the Ouro baseline in the reported downstream table.

Surprising findings

• Explicit CoT prompting on Qwen3-1.7B severely degrades next-token reasoning accuracy (e.g., Easy 47.49 to 10.70).
• Forced-depth accuracy does not improve monotonically with more latent steps, and can degrade for Medium and Hard buckets.

Limitations

• Evaluated only on Ouro-1.4B and Ouro-2.6B, single architecture family.
• Theoretical analysis relies on assumptions (e.g., teacher distribution as label proxy, Lipschitz exit CDF) and only partially validates the method.
• Many downstream gains are small (often under 1 point) without reported variance across seeds.

Experimental setup

Datasets: Omni-Math MMLU MMLU-Pro BBH ARC-C HellaSwag Winogrande GSM8K MBPP MBPP+ HumanEval HumanEval+

Benchmarks: Omni-Math next-token reasoning (easy/medium/hard) MMLU MMLU-Pro BBH ARC-C HellaSwag Winogrande GSM8K MBPP MBPP+ HumanEval HumanEval+

Metrics: accuracy average exit steps pass@1

Baselines: Ouro (vanilla looped) Qwen3-1.7B Qwen3-4B Qwen2.5-3B Gemma3-4B Llama3.2-3B Qwen2.5-7B Qwen3-8B

Builds on: Reinforcement Pre-Training Ouro looped language model GRPO RLVR

Closest papers by meaning

• Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models (ICML 2026 · similarity 0.92)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.89)
• LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation (ICLR 2026 · similarity 0.89)
• RLP: Reinforcement as a Pretraining Objective (ICLR 2026 · similarity 0.89)
• Reinforcement Mid-Training (ICLR 2026 · similarity 0.89)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.88)