Research map / Reinforcement learning / RL from feedback
Self-Improving Robust Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SRPO, an offline preference optimization method that trains a self-improvement policy jointly with the generative policy via a single supervised loss, claiming robustness to the behavior policy and gains from iterative self-revision on summarization.
Problem
Existing RLHF and direct preference optimization methods (PPO, DPO, IPO) depend on the behavior policy used to generate preference data, making them fragile under distribution shift, and their outputs cannot be self-corrected at inference time.
Approach
Casts the min-max objective over a generative policy and an in-context self-improvement policy as a supervised least-squares preference loss, whose solution is independent of the behavior policy; a single LLM serves as both the generator and the reviser, enabling iterative N-revision at inference.
Key results
- On XSum OOD summarization, SRPO 5-rev. reaches the highest win rate against gold summaries, with a larger gap over DPO and IPO than in-distribution.
- On TL;DR in-distribution, SRPO 4-rev. attains the highest win rate against gold summaries, while DPO and IPO do not improve under self-revision.
- On Arena-Hard, SRPO 1-rev. beats DPO 1-rev. with a 67.50% win rate in head-to-head comparison and SimPO 1-rev. with 74.90%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SRPO instead of DPO | LLaMA-3.1-8B Base preference training, 0-rev. · Arena-Hard | head-to-head win rate: 53.67% | Helped |
| Use SRPO instead of IPO | LLaMA-3.1-8B Base preference training, 0-rev. · Arena-Hard | head-to-head win rate: 51.10% | Helped |
| Use SRPO instead of RPO | LLaMA-3.1-8B Base preference training, 0-rev. · Arena-Hard | head-to-head win rate: 51.30% | Helped |
| Use SRPO instead of SIMPO | LLaMA-3.1-8B Base preference training, 0-rev. · Arena-Hard | head-to-head win rate: 63.1% | Helped |
| Use SRPO instead of DPO | LLaMA-3.1-8B Base preference training, 1-rev. · Arena-Hard | head-to-head win rate: 67.50% | Helped |
| Use SRPO instead of IPO | LLaMA-3.1-8B Base preference training, 1-rev. · Arena-Hard | head-to-head win rate: 57.50% | Helped |
| Use SRPO instead of RPO | LLaMA-3.1-8B Base preference training, 1-rev. · Arena-Hard | head-to-head win rate: 58.30% | Helped |
| Use SRPO instead of SIMPO | LLaMA-3.1-8B Base preference training, 1-rev. · Arena-Hard | head-to-head win rate: 74.90% | Helped |
| Add iterative self-improvement revisions | SRPO · TL;DR test split | win rate against gold summaries | Helped |
| Add iterative self-improvement revisions | SRPO · XSum test split | win rate against gold summaries | Helped |
| Add one self-improvement revision | DPO and IPO · TL;DR test split | win rate against gold summaries | No clear effect |
| Add one self-improvement revision | DPO and IPO · XSum test split | win rate against gold summaries | No clear effect |
Surprising findings
- DPO and IPO fail to improve their outputs through self-revision, while SRPO improves consistently with each revision iteration.
- On Arena-Hard, performance of all models drops noticeably after just one revision, except that SRPO 1-rev. remains the best.
Limitations
- Experiments are limited to relatively simple language tasks (summarization, Arena-Hard), not challenging multi-task benchmarks.
- Robustness claim is supported mainly by a synthetic bandit example and one OOD summarization transfer setting.
- Hyperparameters for IPO were not tuned from original paper; the paper notes the baseline comparison relies on chosen settings.
Experimental setup
Datasets: Reddit TL;DR summarization (SFT and preference splits) XSum test split ShareGPT-Vicuna UltraFeedback binarized Arena-Hard
Benchmarks: Reddit TL;DR summarization test XSum test (OOD) Arena-Hard
Metrics: win rate vs gold reference win rate vs LLaMA-3.1-8B-Instruct bootstrap st.dev.
Baselines: DPO IPO SimPO RPO
Builds on: DPO IPO KL-regularized RL Nash-MD
Closest papers by meaning
- Robust Reinforcement Learning from Corrupted Human Feedback (NeurIPS 2024 · similarity 0.89)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.89)
- Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.89)
- Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer (NeurIPS 2024 · similarity 0.89)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.89)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.