Research map / Reinforcement learning / RL from feedback
Conditional Equivalence of DPO and RLHF: Assumptions, Failure Modes, and Provable Alignment
Zhiqin Yang, Yonggang Zhang, Wei Xue, Dong Fang, Bo Han, Yike Guo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adaptive reference-based margin (CPO) to DPO | DPO, Llama-3-8B-Instruct · AlpacaEval 2 | LC win rate (%): 25.09 -> 26.57 | Improved |
| add adaptive reference-based margin (CPO) to DPO | DPO, Llama-3-8B-Instruct · Arena-Hard | win rate (%): 28.9 -> 32.6 | Improved |
| use CPO instead of DPO when the reference policy is misaligned (R=0.2) | DPO, corrupted SFT reference · AlpacaEval 2 (R=0.2) | LC win rate (%): 17.23 -> 27.60 | Improved |
| use CPO instead of DPO when the reference policy is misaligned (R=0.4) | DPO, corrupted SFT reference · AlpacaEval 2 (R=0.4) | LC win rate (%): 15.98 -> 24.34 | Improved |
| increase reference corruption ratio from 0.2 to 0.4 (ablation) | CPO · AlpacaEval 2 | LC win rate (%): 27.60 -> 24.34 | Hurt |
| raise margin weight gamma from 0.15 to 0.20 (ablation) | CPO · AlpacaEval 2 (gpt-4.1 annotator) | LC win rate (%): 26.32 -> 32.56 | Improved |
| vary gamma within 0.2 to 0.4 (ablation) | CPO · AlpacaEval 2 (gpt-4.1 annotator) | WR and LC: WR 26-28%, LC 31-34% | No clear effect |
| replace the adaptive margin with clipping the reference log-ratio to non-negative (ablation) | CPO · AlpacaEval 2 (gpt-4.1 annotator) | LC win rate (%): 33.97 -> 23.86 | Hurt |
| use CPO instead of DPO | DPO, Llama-3-8B-Instruct · IFEval | strict accuracy (%): 34.01% -> 35.12% | Improved |
| use CPO instead of DPO | DPO, Llama-3-8B-Instruct · IFEval | loose accuracy (%): 40.67% -> 43.99% | Improved |
Closest papers by meaning
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.92)
- As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.92)
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML 2024 · similarity 0.92)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
- DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.91)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
