Research map / Reinforcement learning / RL from feedback
Training Language Models to Self-Correct via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCoRe, a two-stage multi-turn on-policy RL method with reward shaping that trains a single Gemini model to self-correct intrinsically on MATH and HumanEval.
Problem
Intrinsic self-correction in LLMs is largely ineffective; prior fine-tuning approaches (STaR, SFT on synthetic repair pairs) suffer from behavior collapse and distribution shift, and require multiple models or oracle supervision.
Approach
Train a single model with on-policy multi-turn REINFORCE in two stages: Stage I decouples attempts by applying a strict KL penalty on the first attempt while optimizing second-attempt reward; Stage II optimizes both attempts with a reward-shaping bonus proportional to the change in correctness (alpha times the difference), which penalizes flipping correct answers to incorrect.
Key results
- On MATH, SCoRe reaches 60.0% Acc@t1 and 64.4% Acc@t2, giving a 4.4% self-correction delta, versus -11.2% for the base model.
- On HumanEval, SCoRe achieves a 12.2% self-correction delta (Acc@t2 64.6%), about 9% above the base model.
- With a 32-solution budget, combining parallel sampling with one round of self-correction yields a 10.5% gain versus 7.4% for parallel sampling alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SCoRe instead of the base model | Gemini 1.5 Flash · MATH500 | Δ(t1, t2): -11.2% -> 4.4% | Helped |
| Apply SCoRe instead of the base model | Gemini 1.0 Pro · HumanEval | Δ(t1, t2): 3.0% -> 12.2% | Helped |
| Apply SCoRe instead of the base model | Gemini 1.0 Pro · MBPP-R | accuracy: 47.3% -> 60.6% | Helped |
| Remove multi-turn training (ablation) | SCoRe · MATH500 | Δ(t1, t2): 4.4% -> -2.4% | Hurt |
| Add multi-turn training (ablation) | SCoRe · MATH500 | Accuracy@t1: 61.8% -> 60.0% | Hurt |
| Remove Stage I (ablation) | SCoRe · MATH500 | Δ(t1, t2): 4.4% -> 2.2% | Hurt |
| Add Stage I (ablation) | SCoRe · MATH500 | Accuracy@t2: 61.4% -> 64.4% | Helped |
| Remove reward shaping (ablation) | SCoRe · MATH500 | Δ(t1, t2): 4.4% -> 2.6% | Hurt |
| Add reward shaping (ablation) | SCoRe · MATH500 | Accuracy@t2: 62.6% -> 64.4% | Helped |
| Replace REINFORCE with STaR in Stage II (ablation) | SCoRe · MATH500 | Accuracy@t2: 64.4% -> 58.4% | Hurt |
| Replace REINFORCE with STaR in Stage II (ablation) | SCoRe · MATH500 | Δ(t1, t2): 4.4% -> 2.2% | Hurt |
| Add correct-to-correct examples to STaR training data (ablation) | STaR · MATH | Δ(t1, t2): -14.2% -> 0.4% | No clear effect |
Surprising findings
- Standard multi-turn RL that optimizes only second-attempt reward still collapses to non-correcting behavior, with Delta(t1,t2) not improving despite higher per-attempt accuracy.
- Replacing REINFORCE with STaR in Stage II gives lower absolute accuracy with no self-correction improvement.
Limitations
- Only one round of self-correction (two attempts) was trained due to infrastructure constraints.
- Stage I and Stage II require separate runs; unifying them is left to future work.
- Results rely on proprietary Gemini models, limiting reproducibility.
Experimental setup
Datasets: MATH MBPP HumanEval
Benchmarks: MATH500 HumanEval MBPP-R
Metrics: Accuracy@t1 Accuracy@t2 Delta(t1,t2) Delta i->c Delta c->i
Baselines: Self-Refine STaR Pair-SFT Base model
Builds on: STaR REINFORCE Self-Refine Pair-SFT
Closest papers by meaning
- Self-Adapting Language Models (NeurIPS 2025 · similarity 0.91)
- A Theoretical Understanding of Self-Correction through In-context Alignment (NeurIPS 2024 · similarity 0.90)
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.90)
- Learning Self-Correction in Vision–Language Models via Rollout Augmentation (ICML 2026 · similarity 0.90)
- Language Model Self-improvement by Reinforcement Learning Contemplation (ICLR 2024 · similarity 0.89)
- Behavior Injection: Preparing Language Models for Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.