Research map / Large language models / Alignment and preferences
Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates the DPO loss into an optimizer and a label-independent regularizer, shows standard DPO oversimplifies the regularizer (causing likelihood underdetermination), and proposes PRO, a unified loss for pairwise, binary and scalar feedback using a hyper-response approximation.
Problem
Direct alignment methods like DPO suppress absolute likelihoods of example responses (likelihood underdetermination), leading to reward-hacking-like behavior such as length exploitation, and existing methods do not unify pairwise, binary and scalar feedback.
Approach
Decomposes the DPO loss into a pointwise optimizer term and a label-independent regularizer; restoring the full regularizer resolves underdetermination, and a hyper-response (aggregating all unobserved responses into one unit) makes it tractable, yielding PRO for all feedback types.
Key results
- On Mistral-7B-sft with UltraFeedback, PRO-B achieves the best average rank (1.7) with 20.46% LC and 21.48% WR on AlpacaEval 2, versus DPO at 18.36% LC / 19.80% WR.
- On Pythia-6.9B with Anthropic-HH, DPO's response length grows sharply and win rate drops, while PRO-P and PRO-B remain stable over training.
- On the 1%-desired imbalanced binary setting, increasing alpha to 17.5 raises PRO-B win rate from 5.57% to 57.21%, above the full-data 53.37%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PRO-P or PRO-B instead of DPO for pairwise or binary alignment | Mistral-7B-sft · UltraFeedback, AlpacaEval 2 WR (%) | WR (%): DPO 19.80; PRO-B 21.48 | Helped |
| Use PRO-P or PRO-B instead of NCA | Mistral-7B-sft · UltraFeedback, MT-Bench | Score: NCA 6.03; PRO-P 6.06; PRO-B 6.06 | Helped |
| Use PRO-B instead of DPO | Mistral-7B-sft · UltraFeedback, IFEval | Score: DPO 19.22; PRO-B 30.13 | Helped |
| Use PRO-P instead of DPO | Mistral-7B-sft · UltraFeedback, ARC | Score: DPO 61.77; PRO-P 61.26 | Hurt |
| Use PRO-B instead of KTO | Mistral-7B-sft · UltraFeedback, GPQA | Score: KTO 33.04; PRO-B 32.81 | Hurt |
| Use PRO-P instead of DPO during alignment | Pythia-6.9B · Anthropic-HH | Reward-hacking symptoms (response length and win rate) | Helped |
| Increase alpha from 2.5 to 17.5 (ablation) | PRO-B on Pythia-6.9B · Anthropic-HH 1%-desired | Win rate (%): 5.57 -> 57.21 | Helped |
| Increase alpha from 2.5 to 10 (ablation) | PRO-B on Pythia-6.9B · Anthropic-HH 1%-undesired | Win rate (%): 48.87 -> 50.64 | Helped |
| Increase responses per instruction from N=2 to N=4 (ablation) | PRO-S on Mistral-7B-sft · UltraFeedback scalar, TruthfulQA | Score: 45.90 -> 49.45 | Helped |
| Increase responses per instruction from N=2 to N=4 (ablation) | PRO-S on Mistral-7B-sft · UltraFeedback scalar, GPQA | Score: 30.80 -> 32.81 | Helped |
| Increase responses per instruction from N=2 to N=4 (ablation) | PRO-S on Mistral-7B-sft · UltraFeedback scalar, IFEval | Score: 29.31 -> 29.43 | Helped |
Surprising findings
- Standard DPO's likelihood underdetermination stems from an oversimplified regularizer, and restoring it removes the need for extra supervised or regression terms.
- KTO's sigmoid-based loss degrades over training (length rises from 113.6 to 235.6, win rate drops 6.64%), attributed to vanishing gradients from saturation.
Limitations
- The hyper-response mechanism cannot distinguish among the probabilities of unobserved responses, leaving their distribution unregulated.
- PRO introduces an extra hyperparameter alpha that must be tuned, with optimal values varying by setting (e.g., 1%-desired vs 1%-undesired).
Experimental setup
Datasets: Anthropic-HH UltraFeedback
Benchmarks: AlpacaEval 2 MT-Bench ARC IFEval TruthfulQA GPQA
Metrics: win rate length-controlled win rate average response length MT-Bench score accuracy average rank
Baselines: DPO KTO NCA IPO
Builds on: DPO KTO NCA IPO
Closest papers by meaning
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.93)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.93)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.92)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.92)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)
- Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.