Research map / Large language models / Reasoning and chain of thought

RLP: Reinforcement as a Pretraining Objective

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLP, a verifier-free reinforcement pretraining objective that rewards sampled chain-of-thought by its log-likelihood gain on the next token, applied to ordinary text during pretraining.

Problem

Next-token prediction pretraining does not explicitly encourage reasoning before prediction, and prior reinforcement pretraining (RPT) uses sparse binary rewards, entropy-based token filtering, and distilled checkpoints, leaving unclear whether base models learn to think.

Approach

Treat a sampled chain-of-thought before each next token as an action; reward it by the log-likelihood ratio of the observed token under the thought-conditioned model versus an EMA no-think teacher, optimized with group-relative advantages and a clipped surrogate on thought tokens only, applied densely over general text.

Key results

• On Qwen3-1.7B-Base, RLP yields overall average 36.03 vs 30.32 for base and 30.85 for CPT before post-training (about 19% relative over base); after SFT+RLVR, 42.51 vs 39.34 (base+Post) and 39.90 (CPT+Post).
• On Nemotron-Nano-12B-v2, RLP raises overall average from 42.81 to 61.32 with 250M tokens, versus 65.31 for the compute-matched baseline after post-training (68.09 for RLP+Post).
• In token-matched RPT comparison on Qwen3-1.7B, RLP reaches 43.35 avg vs 41.69 for RPT; in flop-matched, 42.86 vs 35.68.

Measured effects reported in the paper

Surprising findings

• RLP on Qwen3-1.7B with 170M tokens outperforms CPT trained on 6B tokens (35x more data), matching FLOPs.
• Adding a KL penalty to a fixed reference reduced performance, so the best setting uses beta=0.

Limitations

• Gains on the Nemotron model's post-trained results are smaller relative to base gains, and comparisons rely on author-run baselines with own post-training recipe.
• Reward applied to only one random token per document in main experiments, so coverage of the full objective is partial.
• Theoretical guarantees are partly informal; the monotonic-improvement argument is heuristic and relies on small-step approximations.

Experimental setup

Datasets: OmniMath OpenThoughts Nemotron-Crossthink ACAD Math-Text Web-Crawl OpenThoughts3 MATH

Benchmarks: GSM8K MATH500 Minerva Math AMC23 AIME25 MMLU MMLU-Pro GPQA

Metrics: Pass@1 average over 8 runs greedy and Pass@1 average over 4 runs overall average accuracy

Baselines: Continuous pretraining (CPT) RPT Qwen3-1.7B-Base Nemotron-Nano-12B-v2 base

Builds on: GRPO RPT (reinforcement pretraining) RLVR

Closest papers by meaning

• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.92)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.91)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.91)
• Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models (NeurIPS 2026 · similarity 0.91)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.90)
• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (NeurIPS 2025 · similarity 0.90)