Research map / Large language models / Alignment and preferences
Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
Wei Chen, Yubing Wu, Junmei Yang, Delu zeng, Qibin Zhao, John Paisley, Min Chen, Zhou Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reward calibration (RC) | DPO · UltraFeedback, Mistral-7B | average score: 24.26 -> 36.44 | Improved |
| add reward calibration (RC) | DDRO · UltraFeedback, Mistral-7B | average score: 16.53 -> 36.34 | Improved |
| add reward calibration (RC) | SimPO, BCE, LSIF, TI-DPO (strong baselines) · UltraFeedback, Mistral-7B | average score: SimPO +0.01, BCE -0.05, LSIF +0.15, TI-DPO +0.00 | No clear effect |
| add reward calibration (RC) | DPO, SimPO, BCE, LSIF, TI-DPO · UltraFeedback, Qwen2.5-7B | average score: DPO +0.48, SimPO +0.26, BCE +1.33, LSIF +0.23, TI-DPO +0.11 | Improved |
| add reward calibration (RC) | DDRO · UltraFeedback, Qwen2.5-7B | average score: 45.25 -> 45.20 | Hurt |
| add reward calibration (RC) | DPO · UltraFeedback, Pythia-2.8B | ARC: 34.04 -> 37.12 | Improved |
| add reward calibration (RC) | SimPO · UltraFeedback, Pythia-2.8B | ARC: 32.59 -> 37.03 | Improved |
| add reward calibration (RC) | CPO · UltraFeedback, Pythia-2.8B | MMLU-Pro | Hurt |
| add reward calibration (RC) | DPO · AlpacaEval 2.0, Pythia-2.8B | LC win rate: 70.56% | Improved |
| add reward calibration (RC) | DPO, BCE · Anthropic-HH, Mistral-7B | average score: DPO 36.04 -> 35.93; BCE 35.99 -> 36.09 | No clear effect |
| vary log-domain EMA momentum from 0.5 to 0.999 (ablation) | RC with DPO · UltraFeedback, Pythia-2.8B | average score: 26.91 (w/o RC) -> 28.43 to 28.95 | No clear effect |
| train for 2 epochs instead of 1 (ablation) | RC with DPO · UltraFeedback, Mistral-7B | average score: 36.84 -> 36.93 | No clear effect |
Closest papers by meaning
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.89)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- Bayesian Preference Learning for Test-Time Steerable Reward Models (NeurIPS 2026 · similarity 0.88)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.88)
- A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement (ICLR 2025 · similarity 0.88)
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
