Research map / Large language models / Reasoning and chain of thought

Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirically shows RL (RLVR vs RLIF) reshapes reasoning-pattern distributions by reranking a sparse set of tokens, and builds a tabular gradient-flow theory of pattern selection explaining RLVR convergence regimes and RLIF's early gain then possible degradation.

Problem

Training dynamics of RL for LLM reasoning are poorly understood; it is unclear how RLVR and internal-feedback RL (RLIF) change reasoning behavior and performance over training.

Approach

Model reasoning as question→reasoning pattern→answer, and analyze RL as optimizing the pattern-selection distribution; combine pattern-level GPT-4o classification and token-rank analysis with a tabular gradient-flow theory giving two RLVR convergence regimes and an RLIF analysis showing early accuracy gain but possible convergence to low-accuracy patterns.

Key results

• Token ranking change ratio after RL stays small: RLVR 5.2%-7.3% and RLIF 6.8%-10.1% across GSM8K, MATH, AIME24 over training steps 20-100.
• RLVR shifts reasoning-pattern distribution toward patterns with higher success rates while RLIF does not specialize, on Qwen2.5-3B trained on MATH.
• Theory: under RLIF with uniform success rates and |R| to infinity, accuracy derivative at initialization is positive with probability 1, yet with probability 0.5 the RLIF argmax pattern is the least accurate one.

Surprising findings

• RLIF can initially improve accuracy but may converge to reasoning patterns with low success rates, leaving the model worse than its base model.
• Per-pattern success rates remain largely stable during RL training, so gains come mainly from reweighting pattern selection.

Limitations

• Theory relies on a tabular policy, a fixed per-pattern success-rate assumption, and beta=0, which may not hold for real LLMs.
• Pattern extraction and classification depend on GPT-4o and the interpretability of LLM-identified patterns is not fully validated.
• Large-model results for QwQ-32B cannot report pattern accuracy because only API access was available.

Experimental setup

Datasets: MATH GSM8K AIME24 AMC23 OlympiadBench

Benchmarks: MATH GSM8K AIME24 AMC23 OlympiadBench

Metrics: accuracy pattern distribution shift token rank change ratio pattern success rate

Baselines: RLIF (self-certainty / entropy-style internal reward) Qwen2.5-7B-SimpleRL-Zoo QwQ-32B

Builds on: GRPO verl SimpleRL-Zoo

Closest papers by meaning

• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.95)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.94)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.94)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.94)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.94)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.93)