Research map / Large language models / Reasoning and chain of thought
RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that self-generated positive synthetic data beats oracle positives, and that per-step DPO on critically-chosen negative responses (equivalent to advantage-weighted RL) yields up to 8x sample-efficiency gains for math reasoning.
Problem
Synthetic data for LLM math reasoning helps unevenly: positive synthetic data from capable models scales slowly and self-generated positives can amplify spurious intermediate steps. The question is when synthetic data (positive and negative) helps or hurts.
Approach
Construct learner-specific negative pairs that localize the first critical step (lowest Monte Carlo Q-value under a best-of-K policy) and train with per-step DPO, which the authors show is equivalent to advantage-weighted RL; this unlearns spurious steps that positive-only SFT/RFT amplifies.
Key results
- Per-step DPO on negatives gives performance comparable to scaling positive synthetic data by 8x, versus about 2x for RFT.
- Positive data scaling follows roughly D^-0.15 on GSM8K and D^-0.05 on MATH.
- Standard DPO with random positive/negative pairing does not improve over the SFT policy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale positive synthetic problems and solutions from GPT-4 and Gemini 1.5 Pro | SFT · GSM8K | test error rate: D^{-0.15} | Helped |
| Scale positive synthetic problems and solutions from GPT-4 and Gemini 1.5 Pro | SFT · MATH | test error rate: D^{-0.05} | Helped |
| Use self-generated verified positive responses instead of one teacher response per problem | RFT · GSM8K and MATH | sample efficiency: 2× | Helped |
| Use one self-generated positive response per problem (ablation) | RFT · GSM8K and MATH | test performance: >1% | Helped |
| Scale self-generated positive responses that may contain spurious steps (ablation) | RFT · GSM8K | test error rate | Hurt |
| Add learner-generated negative responses with critical-step credit assignment | Per-step DPO · GSM8K and MATH | sample efficiency: 8× | Helped |
| Pair arbitrary positive and negative responses rather than identify critical steps (ablation) | Standard DPO · GSM8K and MATH | test performance | Hurt |
| Pair positive and negative responses with highest edit distance rather than identify critical steps (ablation) | DPO · GSM8K and MATH | test performance | Mixed |
| Add preference pairs from positive traces to per-step DPO pairs from negative traces (ablation) | Per-step DPO · GSM8K and MATH | test error | Helped |
| Generate negative data from an under-trained SFT initialization (ablation) | Per-step DPO · Didactic star graph problem | test performance | Helped |
| Generate negative data from an early-stopped SFT initialization (ablation) | Per-step DPO · Didactic star graph problem | test performance | Helped |
| Generate negative data from an over-trained SFT initialization (ablation) | Per-step DPO · Didactic star graph problem | test error | No clear effect |
Surprising findings
- Self-generated positive responses from the 7B SFT policy are as effective as data from GPT-4/Gemini with 2x the problems.
- Scaling positive self-generated RFT data can saturate or worsen test error because of spurious steps that the answer verifier cannot detect.
Limitations
- Synthetic data from stronger models may contain errors on harder reasoning problems, which is not fully addressed.
- Advantage estimates via few Monte Carlo rollouts can overestimate advantages, causing failure on over-trained SFT checkpoints.
- Evaluation limited to GSM8K and MATH without transfer evaluation.
Experimental setup
Datasets: GSM8K MATH
Benchmarks: GSM8K MATH
Metrics: test error rate pass@5 error rate scaling law exponent
Baselines: SFT RFT standard DPO per-step DPO (Q-value pairs) edit-distance paired DPO
Builds on: direct preference optimization rejection finetuning STaR Group DRO per-step DPO
Closest papers by meaning
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.90)
- Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data (AAAI 2024 · similarity 0.90)
- NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning (ICLR 2026 · similarity 0.90)
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.89)
- Are Reasoning LLMs Robust to Interventions on their Chain-of-Thought? (ICLR 2026 · similarity 0.89)
- AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.