Research map / Learning theory and algorithms / Prediction and decision losses
On Extending Direct Preference Optimization to Accommodate Ties
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPO-RK and DPO-D, two DPO variants that replace Bradley-Terry with tie-aware Rao-Kupper and Davidson models so that tied preference pairs can be used in training without the performance loss seen with plain DPO, and shows added regularization against the reference policy.
Problem
Standard DPO assumes every preference pair is a clear win/loss and discards tied pairs; adding tied pairs to DPO degrades task performance, so the informative tie data is wasted.
Approach
Replace the Bradley-Terry preference model in DPO with the Rao-Kupper or Davidson tie-aware extensions, giving a ternary (win/loss/tie) likelihood so tied pairs can be trained on directly; the ties act as a regularizer that keeps the policy close to the reference model.
Key results
- On ALMA-R-Preference ZH-EN, DPO-RK(CP+TP) reaches 90.40 XCOMET and 75.77 KIWI-XXL versus 88.87 and 74.12 for DPO(CP).
- On GSM8K with Qwen2.5-3B-Instruct, DPO-D(CP+TP) reaches 84.5% at beta 0.5-0.7 versus 83.7% for DPO(CP) at its best beta, with lower KL (1.465 vs 2.258).
- Adding tied pairs to plain DPO hurts task performance across NMT and summarization, while DPO-RK and DPO-D recover DPO(CP)-level performance at smaller KL values.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add conflicting-metric pairs as ties and use DPO-RK instead of DPO on clear preference pairs | ALMA-7B-LoRA · ALMA-R-Preference ZH-EN test | COMET: 79.66 -> 80.63 | Helped |
| Add conflicting-metric pairs as ties and use DPO-D instead of DPO on clear preference pairs | ALMA-7B-LoRA · ALMA-R-Preference ZH-EN test | COMET: 79.66 -> 80.38 | Helped |
| Include tied pairs in standard DPO training (ablation) | DPO · NMT and TL;DR | task performance | Hurt |
| Use a tie-compatible preference model (Rao-Kupper or Davidson) with tied pairs | DPO · NMT and TL;DR | task performance | Helped |
| Add ties to preference optimization | DPO-RK and DPO-D · GSM8K test | exact match: 83.7% -> 84.4%; 83.7% -> 84.5% | Helped |
| Add tied pairs to training | DPO-RK and DPO-D · GSM8K test | KL: 2.258 -> 1.762; 2.258 -> 1.465 | Helped |
| Add tied pairs to training | DPO-RK and DPO-D · GSM8K test | PR: 95.19% -> 97.11%; 95.19% -> 97.65% | Helped |
| Include tied pairs in standard DPO rather than filtering them (ablation) | DPO · ZH-EN translation test | Mean(XCOMET and KIWI-XXL): 83.2 -> 82.9 | Hurt |
| Use DPO-D with tied pairs instead of DPO on clear preference pairs | ALMA-7B-LoRA · ZH-EN translation test | Mean(XCOMET and KIWI-XXL): 83.2 -> 83.4 | Helped |
| Add tied pairs to training data | DPO-RK and DPO-D · WMT18 ZH-EN held-out preference classification | tied-pair classification accuracy: +30% | Helped |
| Use lower β during training | Rao-Kupper classifier · WMT18 ZH-EN held-out preference classification | overall accuracy: +10% overall Acc. from β=1.0 to β=0.1 | Helped |
Surprising findings
- Plain DPO trained with tied pairs assigns extreme preference probabilities (near 0 or 1) to held-out ties, whereas the tie-aware variants keep tie margins near zero.
- Including tied pairs regularizes DPO toward the reference model even for the original DPO formulation, without improving task performance.
Limitations
- No human-annotated tie data is available; ties are constructed heuristically from automatic metrics.
- The nu hyperparameters (nu_RK=3, nu_D=1) were set by motivation rather than tuned, so better performance may be possible with tuning.
- Results are limited to translation, summarization, and math reasoning with a few 3B-7B models.
Experimental setup
Datasets: WMT20 ZH-EN WMT21 ZH-EN IWSLT17 FR-EN TL;DR ALMA-R-Preference GSM8K WMT18 ZH-EN
Benchmarks: WMT21 ZH-EN IWSLT17 FR-EN TL;DR GSM8K ALMA-R-Preference test set
Metrics: BLEURT COMET KIWI-22 XCOMET KIWI-XXL PairRM win-rate KL divergence exact match accuracy preservation rate preference classification accuracy
Baselines: DPO IPO SimPO CPO ALMA-7B-R
Builds on: DPO Bradley-Terry model Rao-Kupper model Davidson model ODPO DPO-ST
Closest papers by meaning
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.89)
- Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization (ICLR 2025 · similarity 0.88)
- Preference Optimization by Estimating the Ratio of the Data Distribution (NeurIPS 2025 · similarity 0.88)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.88)
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.88)
- Understanding the Logic of Direct Preference Alignment through Logic (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.