Research map / Large language models / Alignment and preferences
Iterative Reasoning Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Iterative RPO, an iterative DPO+NLL preference optimization scheme that builds winning-vs-losing chain-of-thought pairs from answer correctness on fixed training sets, improving Llama-2-70B-Chat on GSM8K, MATH, and ARC-Challenge.
Problem
Iterative preference optimization helps general instruction following but yields little gain on chain-of-thought reasoning tasks.
Approach
Sample multiple CoT solutions per training problem, build preference pairs by final-answer correctness, and train with a DPO loss plus a length-normalized NLL term on the winning response, iterating with the updated model as both generator and reference.
Key results
- GSM8K test accuracy rises from 55.6% zero-shot CoT to 81.6% after iteration 4 (88.7% with majority voting over 32 samples).
- ARC-Challenge accuracy reaches 86.7% after iteration 3, versus 77.8% zero-shot CoT and 83.5% for standard DPO.
- MATH accuracy improves from 12.5% few-shot CoT to 20.8% after iteration 3 (29.1% with majority voting).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Iterate preference optimization from the previous model | Llama-2-70B-Chat with Iterative RPO · GSM8K test | accuracy: 55.6% to 81.6% | Helped |
| Use majority voting over 32 samples | Iterative RPO · GSM8K test | accuracy: 81.6% to 88.7% | Helped |
| Use Iterative RPO instead of zero-shot CoT | Llama-2-70B-Chat · ARC-Challenge test (0-shot) | accuracy: 77.8% to 86.7% | Helped |
| Use majority voting over 32 samples | Iterative RPO · ARC-Challenge test (0-shot) | accuracy: 86.7% to 87.9% | Helped |
| Use Iterative RPO instead of few-shot CoT | Llama-2-70B-Chat · MATH test (4-shot) | accuracy: 12.5% to 20.8% | Helped |
| Use majority voting over 32 samples | Iterative RPO · MATH test (4-shot) | accuracy: 20.8% to 29.1% | Helped |
| Add an NLL term to standard DPO (ablation) | DPO training on GSM8K preference pairs · GSM8K test | accuracy: 61.8% to 73.1% | Helped |
| Add an NLL term to DPO (ablation) | DPO training on ARC-Challenge preference pairs · ARC-Challenge test (0-shot) | accuracy: 82.8% to 84.8% | Helped |
| Add an NLL term to DPO (ablation) | DPO training on MATH preference pairs · MATH test (4-shot) | accuracy: 12.4% to 17.7% | Helped |
| Use twice as much paired data in one Iterative RPO iteration (ablation) | Iterative RPO · GSM8K test | accuracy: 73.1% to 74.8% | Helped |
| Train for a second Iterative RPO iteration rather than doubling data in one iteration (ablation) | Iterative RPO · GSM8K test | accuracy: 74.8% to 78.0% | Helped |
| Use twice as much data for one STaR iteration (ablation) | STaR · GSM8K test | accuracy: 65.2% to 66.9% | Helped |
Surprising findings
- Standard DPO on the same preference pairs performs no better than SFT on chosen sequences, and on MATH DPO degrades performance below initialization (12.4% vs 12.5% CoT baseline; 10.5% from SFT init).
- Log-probabilities of chosen sequences decrease under DPO without NLL but increase with the NLL term, which the authors link to the accuracy gains.
Limitations
- Requires gold final answers and an exact-match correctness measure, so it does not directly apply to open-ended tasks.
- Gains decay across iterations and saturate, since the fixed training prompts are reused.
- Training at iteration t does not use data from earlier iterations; on ARC, noise from lucky guesses in multiple-choice tasks may add noise to CoT pairs.
Experimental setup
Datasets: GSM8K MATH ARC-Challenge
Benchmarks: GSM8K MATH ARC-Challenge
Metrics: exact match accuracy majority voting accuracy over 32 samples
Baselines: Zero-shot CoT SFT on gold CoT DPO STaR SFT on chosen sequences Iterative DPO MetaMath WizardMath
Builds on: DPO Self-Rewarding LLMs STaR Iterative DPO SPIN
Closest papers by meaning
- Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization (ICLR 2026 · similarity 0.89)
- Preference Optimization for Reasoning with Pseudo Feedback (ICLR 2025 · similarity 0.89)
- Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs (NeurIPS 2024 · similarity 0.88)
- Building Math Agents with Multi-Turn Iterative Preference Learning (ICLR 2025 · similarity 0.88)
- Advancing LLM Reasoning Generalists with Preference Trees (ICLR 2025 · similarity 0.88)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.