Research map / Large language models / Reasoning and chain of thought

On Predictability of Reinforcement Learning Dynamics for Large Language Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that the top singular subspace (Rank-1) of RL-induced parameter updates recovers most reasoning gains and evolves linearly over training, and uses this to propose AlphaRL, an extrapolation method that accelerates RL training up to 2.5x.

Problem

Parameter dynamics of RL training for LLMs are poorly understood; it is unclear whether RL-induced weight updates follow consistent low-dimensional principles that could make training trajectories predictable.

Approach

RL weight updates are dominated by their rank-1 SVD component, whose trajectory over training is approximately linear, so the final update can be extrapolated from a short early window of checkpoints (AlphaRL), with no extra modules or tuning.

Key results

• Rank-1 update recovers on average 99.17% of reasoning performance of fully trained RL models across tested models and algorithms.
• AlphaRL retains over 96% of final reasoning performance (RLOO and GRPO on Qwen3-8B reach ~96% relative accuracy) with up to 2.5x speedup.
• Linear fit of Rank-1 trajectory yields average R^2 of 0.914 for RL-trained models, while SFT and distillation do not show the property.

Surprising findings

• Rank-1 updates alone can match or surpass the fully trained RL model on several benchmarks (e.g., RLOO, GRPO, DAPO on MATH-500).
• SFT and distillation updates are one to two orders of magnitude larger in norm than RL updates and do not exhibit rank-1 dominance.

Limitations

• Lacks rigorous theoretical foundation; conclusions are empirical.
• AlphaRL effectiveness is constrained by RL algorithm design and stability.
• Linear extrapolation assumed; nonlinear forecasting not explored.

Experimental setup

Datasets: DAPO-Math-17k DeepMath-103K Math-Step-DPO-10K Kuhn Poker

Benchmarks: AIME24 AIME25 MATH-500 MINERVA GPQA GSM8K

Metrics: accuracy (average over 32 samples, T=0.6) relative accuracy R^2 of PLS linear fit

Baselines: SFT DIST DPO On-Policy Distillation Spiral

Builds on: GRPO DAPO RLOO Dr.GRPO PPO LlamaFactory

Closest papers by meaning

• Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.93)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.92)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.92)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.92)
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.91)