Research map / Reinforcement learning / RL from feedback
Displacement-Resistant Extensions of DPO with Nonconvex $f$-Divergences
Idan Pipano, Shoham Sabach, Kavosh Asadi, Mohammad Ghavamzadeh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SquaredPO (non-convex f-DPO with adaptive beta) instead of DPO | Llama-3-8B-Instruct · TL;DR validation vs reference (epoch 4) | win rate: 34.7 -> 51.0 | Improved |
| use SquaredPO instead of DPO | Llama-3-8B-Instruct · TL;DR validation vs reference (epoch 2) | win rate: 45.0 -> 50.6 | Improved |
| use SquaredPO instead of DPO | Llama-3-8B-Instruct · TL;DR validation vs reference (epoch 1) | win rate: 51.8 -> 50.8 | No clear effect |
| use SquaredPO instead of chiPO | Llama-3-8B-Instruct · TL;DR validation vs reference (epoch 4) | win rate: 48.3 -> 51.0 | Improved |
| use SquaredPO instead of DPO | Llama-3-8B-Instruct · AlpacaEval 2 | length-controlled win rate: 29.6 -> 29.2 | No clear effect |
| use SquaredPO instead of DPO | Llama-3-8B-Instruct · AlpacaEval 2 | win rate: 24.8 -> 24.5 | No clear effect |
| use SquaredPO instead of DPO | Llama-3-8B-Instruct · MT-Bench | score: 7.925 -> 7.924 | No clear effect |
| use SquaredPO instead of chiPO | Llama-3-8B-Instruct · MT-Bench | score: 7.900 -> 7.924 | Improved |
| use SquaredPO instead of chiPO | Llama-3-8B-Instruct · AlpacaEval 2 | length-controlled win rate: 29.3 -> 29.2 | No clear effect |
| use SquaredPO instead of DPO | Llama-3-8B-Instruct + TL;DR · monotone decrease of chosen log-ratios over 4 epochs | percent of winners: 99.63 -> 4.21 | Improved |
| use SquaredPO instead of DPO | Qwen2-0.5B-Instruct + UltraFeedback · monotone decrease of chosen log-ratios over 4 epochs | percent of winners: 35.57 -> 1.64 | Improved |
| use SquaredPO instead of DPO | Llama-3-8B-Instruct + TL;DR · decrease from epoch 1 to 2 | percent of winners: 99.99 -> 10.91 | Improved |
Closest papers by meaning
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.88)
- Preference Optimization by Estimating the Ratio of the Data Distribution (NeurIPS 2025 · similarity 0.87)
- DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.87)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.86)
- Theoretical guarantees on the best-of-n alignment policy (ICML 2025 · similarity 0.86)
- One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
