Research map / Reinforcement learning / RL from feedback

ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ProRL, a prolonged RL recipe (GRPO+DAPO tweaks, KL penalty, periodic reference-policy reset) over 136K multi-domain verifiable tasks, yielding a 1.5B reasoning model that expands pass@k boundaries over its DeepSeek-R1-Distill-Qwen-1.5B base.

Problem

Whether RL genuinely expands LLM reasoning capabilities beyond the base model's distribution, and whether prolonged RL compute reliably improves reasoning, given prior pass@k findings that RL only sharpens existing outputs.

Approach

Extend RL training to thousands of steps using a KL penalty against a periodically hard-reset reference policy, DAPO-style clip-higher and dynamic sampling, and a broad multi-domain verifiable task mix, so that the model discovers solutions the base model cannot reach under extensive sampling.

Key results

• Nemotron-Research-Reasoning-Qwen-1.5B averages 60.14 pass@1 on math versus 44.45 for DeepSeek-R1-Distill-Qwen-1.5B, and 37.49 vs 23.08 on code.
• On Reasoning Gym logic puzzles the average rises from 4.24 to 59.06, and GPQA Diamond from 15.86 to 41.78.
• Pass@k analysis shows gains are largest on tasks where the base model is weakest; some tasks (e.g., boxnet, family_relationships) move from near-zero to solved.

Measured effects reported in the paper

Surprising findings

• Pass@128 often declines on math tasks after RL despite pass@1 gains, indicating sharpening rather than expansion on those tasks.
• Response length first decreases and then increases with validation improvements, and length is not decisive for performance.

Limitations

• Experiments are limited to a 1.5B model, so scalability to larger models is unclear.
• Periodic hard resets of reference policy and optimizer add complexity and may give inconsistent results.
• Training requires substantial compute, limiting accessibility.

Experimental setup

Datasets: DeepScaleR Dataset Eurus-2-RL Dataset SCP-116K Reasoning Gym Llama-Nemotron post-training dataset PRIME validation set AIME2024 AIME2025 AMC MATH Minerva Math OlympiadBench

Benchmarks: AIME2024 AIME2025 AMC MATH Minerva Math OlympiadBench APPS CodeContests Codeforces TACO HumanEval+ LiveCodeBench

Metrics: pass@1 pass@16 pass@128 pass@256 creativity index

Baselines: DeepSeek-R1-Distill-Qwen-1.5B DeepSeek-R1-Distill-Qwen-7B DeepScaleR-1.5B DeepCoder-1.5B

Builds on: GRPO DAPO DeepScaleR verl

Closest papers by meaning

• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.94)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.93)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.93)
• LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts (ICLR 2026 · similarity 0.92)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.92)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.92)