Research map / Large language models / Reasoning and chain of thought

Scheduling Your LLM Reinforcement Learning with Reasoning Trees

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Re-Schedule, an RLVR data scheduling method that ranks queries by a tree-structured Reasoning Score (r-score) and runs an easy-to-hard curriculum via GRPO-style loss weighting.

Problem

Data scheduling for RLVR on math reasoning typically ranks queries by path-based accuracy, which misjudges true learning difficulty; the paper asks how to measure difficulty from the structure of each query's reasoning tree.

Approach

Approximate each query's reasoning tree offline by branching sampled generations every l tokens (k-ary, depth d), define an r-score as the maximum accuracy gain achievable by editing up to M nodes, and use it as a time-varying weight so training moves from high-r-score (structurally simple) to low-r-score queries.

Key results

• On Qwen2.5-Math-7B, Re-Schedule (sigmoid) reaches 48.5 average accuracy across six math benchmarks vs. 46.6 for ACC_sigmoid and 44.3 for GRPO.
• On Qwen2.5-7B, Re-Schedule (sigmoid) reaches 44.5 average vs. 41.3 for ACC_sigmoid and 40.7 for GRPO.
• Reverse (hard-to-easy) schedule drops average accuracy from 48.3 to 44.7 with sigmoid mapping, supporting easy-to-hard ordering.

Surprising findings

• Trees built with coarser token intervals (l=1200) degrade performance substantially (46.0 vs 48.3 average on Qwen2.5-Math-7B).
• Dynamically recomputing r-scores three times during training gives only marginal gains over the static r-score (48.9 vs 48.3), so the static prior is treated as sufficient.

Limitations

• Reasoning tree is an offline approximation built from sampled trajectories, with substantial preprocessing cost (up to ~48.5% extra training time for the 4^4 tree).
• Main experiments focus on math reasoning; code generation evaluated only with a single small model and one dataset.
• Headline gains are modest (up to 3.2 points) and the inconsistent ablation tables report different numbers for the same default configuration.

Experimental setup

Datasets: DAPO-Math-17k ArcherCodeR

Benchmarks: AIME24 AIME25 AMC23 MATH500 Minerva Math OlympiadBench LiveCodeBench v5

Metrics: avg@32 accuracy pass@1 pass@4

Baselines: GRPO SimpleRL-Zoo Eurus-PRIME OPO ACC (sigmoid) LPPO Seed-GRPO R^3 DELT

Builds on: GRPO RLVR curriculum learning LIMR verl

Closest papers by meaning

• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.91)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.91)
• Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning (ICLR 2026 · similarity 0.91)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.90)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.90)