Research map / Large language models / Reasoning and chain of thought

On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses a fully controlled synthetic GSM-Infinite-style testbed with 100M Qwen2.5-style models to study how pre-training, mid-training, and GRPO-based RL interact in producing extrapolative and contextual reasoning generalization.

Problem

It is unclear whether RL truly extends a language model's reasoning beyond pre-training, since opaque pre-training corpora, underexplored mid-training, and RL interactions obscure causal effects.

Approach

Train small models from scratch on a controllable DAG-based synthetic arithmetic corpus with disjoint pre-, mid-, and post-training splits, and evaluate with process-verified pass@k to isolate each stage's causal contribution, including a compute-matched mid-training vs. RL allocation study and process-level rewards.

Key results

• RL on operation counts 11-14 improves pass@128 on OOD-hard tasks (up to +42% pass@128 when well-calibrated), while RL on in-distribution tasks never improves pass@128 beyond the base model.
• Contextual transfer to long-tailed context B fails with 0% or 0.1% pre-training exposure but works with >=1% exposure, yielding up to +60% pass@128.
• Mid-training plus RL outperforms RL alone by +10.8% on OOD-hard tasks under a fixed compute budget.

Surprising findings

• RL on in-distribution tasks sharpens pass@1 but yields no pass@128 gain, consistent with prior claims that RL only refines existing capability.
• Process-level reward mixing improves pass@1 by 4-5% on extrapolative tasks, indicating outcome-only rewards are vulnerable to reward hacking.

Limitations

• Experiments use tiny 100M models trained from scratch on synthetic arithmetic data, limiting transfer to real LLM pipelines.
• Mid-training is a simplified continued-pretraining variant rather than a full industrial mid-training recipe.

Experimental setup

Datasets: GSM-Infinite-style synthetic reasoning corpus (30B tokens, custom generated)

Benchmarks: synthetic extrapolative tasks (op 2-10 ID, 11-14 OOD-edge, 15-20 OOD-hard) contextual transfer on held-out templates

Metrics: pass@1 pass@128 process accuracy

Baselines: RL-only training Full mid-training Full RL

Builds on: GSM-Infinite GRPO Physics of Language Models Chinchilla scaling

Closest papers by meaning

• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.95)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.94)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.93)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.93)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.93)
• PRISM: Demystifying Retention and Interaction in Mid-Training (ICML 2026 · similarity 0.93)