Research map / Large language models / Reasoning and chain of thought

Reinforcement Mid-Training

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Reinforcement Mid-Training (RMT), an intermediate stage applying RL with a dynamic token budget, entropy-based curriculum sampling, and a combined RL plus next-token-prediction objective to unlabeled pre-training text, improving language modeling and subsequent math post-training.

Problem

Applying reinforcement learning to large unlabeled pre-training data (mid-training) is computationally expensive, suffers from overthinking, ignores the imbalanced token entropy distribution, and discards low-entropy tokens.

Approach

Formalizes reinforcement mid-training as RL on selected high-entropy key tokens plus next-token prediction on the rest, with a decaying token budget and length reward to curb overthinking, curriculum sampling from easy to hard tokens, and a unified loss L = L_RL + lambda * L_NTP.

Key results

• RMT-Q3 reaches 59.12 average accuracy on OmniMATH language modeling vs 35.85 for RPT (+64.91%), with response length 186 vs 872 tokens.
• After continual post-training on Skywork, RMT-Q3 reaches 64.33 accuracy vs 54.17 for Qwen3-14B (+18.76%).
• RMT-R1 reaches 46.87 average vs 35.85 for RPT (+30.74%) on language modeling.

Surprising findings

• Next-token reasoning (NTR) baselines perform very poorly (average 3-6) compared with standard NTP baselines (around 31-36).

Limitations

• Evaluation limited to a small OmniMATH set (200 eval items) and one math post-training dataset.
• Hyperparameters such as trade-off weights and budget schedule are fixed without reported sensitivity analysis in the main text.
• Baseline comparison limited to RPT and two base models.

Experimental setup

Datasets: OmniMATH Skywork math dataset

Benchmarks: OmniMATH (next-token accuracy, easy/medium/hard) Skywork math (continual post-training accuracy)

Metrics: accuracy average response length generation time

Baselines: R1-Distill-Qwen-14B Qwen3-14B RPT

Builds on: RPT (Reinforcement Pre-Training) GRPO

Closest papers by meaning

• PRISM: Demystifying Retention and Interaction in Mid-Training (ICML 2026 · similarity 0.91)
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.91)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.90)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.90)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.89)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)