Research map / Reinforcement learning / RL from feedback
Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R³, which trains LLMs for reasoning with outcome-only rewards by sliding the start state of RL rollouts backward along correct demonstrations (reverse curriculum), mixing start states across stages, and optimizing with PPO.
Problem
RL for multi-step LLM reasoning suffers from sparse outcome rewards that do not reveal which steps caused errors, while process supervision needs costly step-level annotation and reward models.
Approach
Start RL rollouts from intermediate states of correct demonstrations, moving the start state from the end of the demonstration back to the beginning in stages, so outcome-only rewards give approximately step-level supervision; mixes start states across stages for stability.
Key results
- On eight CoT reasoning tasks with Llama2-7B, R³ averages 65.62 vs 61.48 for RL (+4.1) and 60.19 for SFT (+5.4).
- On GSM8K program-based reasoning, Codellama-7B + R³ reaches 74.2 vs 70.7 for RL and 63.3 for SFT; Llama2 + R³ reaches 68.9 vs 63.1 RL.
- Tora-code + R³ reaches 76.3 vs 72.6 for Tora-code alone on GSM8K P-CoT, using only original data in RL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reverse-curriculum mixed-stage RL instead of standard RL | Llama2-Base-7B · GSM8K CoT | accuracy: 44.67 -> 50.49 | Helped |
| Use reverse-curriculum mixed-stage RL instead of standard RL | Llama2-Base-7B · SVAMP | accuracy: 57.30 -> 64.40 | Helped |
| Use reverse-curriculum mixed-stage RL instead of standard RL | Llama2-Base-7B · BGQA-main | accuracy: 65.50 -> 67.75 | Helped |
| Use reverse-curriculum mixed-stage RL instead of standard RL | Llama2-Base-7B · MNLI | accuracy: 66.15 -> 72.31 | Helped |
| Use reverse-curriculum mixed-stage RL instead of standard RL | Llama2-Base-7B · race@High | accuracy: 61.50 -> 68.50 | Helped |
| Set KL coefficient β to 0 (ablation) | R³ · GSM8K CoT | performance: 50.5 -> 46.6 | Hurt |
| Set KL coefficient β to 0.1 (ablation) | R³ · GSM8K CoT | performance: 50.5 -> 44.1 | Hurt |
| Remove partial reward ε (ablation) | R³ · GSM8K CoT | performance: 50.5 -> 44.6 | Hurt |
| Set partial reward ε to 0.1 or 0.3 (ablation) | R³ · GSM8K CoT | performance: 45.9; 45.4 | Hurt |
| Apply difficulty-based reward functions instead of the original reward (ablation) | R³ · GSM8K, MNLI, race@High | performance: GSM8K: 50.5 -> 43.7, 41.9, 44.1, 45.1; MNLI: 72.3 -> 68.5, 68.5, 70.0, 68.5; race@High: 68.5 -> 65.0, 65.5, 66.0, 66.0 | Hurt |
| Increase the number of curriculum stages to an appropriate value such as 5 or 6 (ablation) | R³ · GSM8K | performance | Helped |
| Remove increasingly difficult data from training (ablation) | R³ · GSM8K, MNLI, race@High, BGQA | accuracy: GSM8K: 48.9, 48.0, 47.2, 46.9, 44.7 vs 50.5; MNLI: 71.5, 69.2, 66.9, 66.9, 65.4 vs 72.3; race@High: 64.5, 64.0, 64.0, 63 | Hurt |
Surprising findings
- Setting the KL coefficient to 0 did not cause collapse for R³, unlike prior RL findings where the model collapses without KL penalty.
- Difficulty-scaled reward functions (linear, square, sqrt, discount) reduced performance relative to the original binary reward.
Limitations
- Experiments limited to 7B-scale models; scaling to larger models left to future work.
- Relies on access to correct demonstrations (at least one per problem) and SFT warm-up before RL.
- Staged RL alone gives only small gains; method requires mixing stages, adding design choices such as number of stages M and partial reward epsilon tuned per dataset.
Experimental setup
Datasets: GSM8K SVAMP BoardgameQA MNLI SNLI RACE CoT-Collection
Benchmarks: GSM8K SVAMP BoardgameQA MNLI SNLI RACE
Metrics: accuracy
Baselines: Few-shot CoT SFT RL (outcome-supervised PPO) Staged RL MAmmoTH-Coder Tora Tora-code GPT-3.5-Turbo GPT-4 Codex
Builds on: PPO reinforcement learning from demonstrations chain-of-thought Alpaca
Closest papers by meaning
- Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models (ICLR 2026 · similarity 0.90)
- Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.90)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
- Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning (ICLR 2026 · similarity 0.89)
- Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.