Research map / Large language models / Alignment and preferences
Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BMC, which synthesizes a pseudo-winning response by minimally editing the losing response using the winning response as reference, and reweights token-level DPO rewards by the policy's confidence, improving DPO on QA, math, and instruction-following tasks.
Problem
Winning and losing responses in pairwise preference data are generated in isolation, giving weak correlation between them; DPO aggregates token rewards uniformly, so it fails to emphasize the nuanced differences that matter for alignment.
Approach
Bridge correlations by rewriting the losing response into a pseudo-winning response that keeps its style but adopts the winning response's correct content, then model token-level differences by scaling DPO token rewards with a stop-gradient of inverse policy probability, capped at a threshold delta.
Key results
- DPO-BMC reaches 65.1 average QA accuracy vs 61.3 for DPO on Llama2-7B and 49.6 vs 48.3 on math tasks.
- On AlpacaEval 2 with Llama3-8B-Base, DPO-BMC gets 22.4 LC win rate vs 16.0 for DPO.
- On Qwen2.5-14B-Base, DPO-BMC improves AlpacaEval 2 LC from 36.3 to 45.1 over DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DPO with BMC, including targeted pseudo-winning synthesis and confidence-based token weighting | Llama2-7B-base · QA tasks average | accuracy: 3.8 absolute points | Helped |
| replace DPO with BMC, including targeted pseudo-winning synthesis and confidence-based token weighting | Llama2-7B-base · math tasks average | accuracy: 1.3 points | Helped |
| use DPO-BMC instead of DPO | Llama3-8B-Base · AlpacaEval 2 | length-controlled win rate (%): 16.0 -> 22.4 | Helped |
| use DPO-BMC instead of DPO | Qwen2.5-14B-Base · AlpacaEval 2 | length-controlled win rate (%): +8.8 | Helped |
| use DPO-BMC instead of DPO | Qwen2.5-14B-Base · Arena-Hard | win rate (%): +7.3 | Helped |
| use constructed preference data with DPO's original objective, removing confidence-based token weighting (ablation) | DPO-BMC · ECQA test | accuracy: 75.9 -> 75.7 | Hurt |
| use constructed preference data with DPO's original objective, removing confidence-based token weighting (ablation) | DPO-BMC · QASC test | accuracy: 63.0 -> 62.0 | Hurt |
| use the proposed token-weighting objective on original preference data, removing targeted pseudo-winning synthesis (ablation) | DPO-BMC · ECQA test | accuracy: 75.9 -> 74.8 | Hurt |
| use the proposed token-weighting objective on original preference data, removing targeted pseudo-winning synthesis (ablation) | DPO-BMC · QASC test | accuracy: 63.0 -> 60.0 | Hurt |
| generate the losing response by continuing a winning-response prefix (DPO (CW)) instead of targeted modification (ablation) | DPO · QA tasks | accuracy | Hurt |
| degenerate the winning response into a pseudo-losing response and use it as the preference pair (ablation) | Bridging Phase · preference data synthesis | performance | Hurt |
| increase data modification from 0% to 20% (ablation) | Bridging Phase · data modification proportion ablation | performance: 0% to 20% | Helped |
Surprising findings
- Reversing the construction (degenerating the winning response into a pseudo-losing response) hurts performance, as LLMs struggle to produce plausible low-quality outputs.
- Using the open-source Llama3-70B-Instruct for targeted modification gives comparable results to gpt-4-0125-preview.
Limitations
- Relies on a strong LLM (gpt-4-0125-preview) for targeted modification, with API cost.
- Requires filtering pairs where the SFT output is already close to the ground truth, discarding 31% to 43% of data in QA and math.
- Gains on Arena-Hard for Llama3-8B are small (+0.5 WR over DPO).
Experimental setup
Datasets: ECQA QASC MetaMathQA UltraFeedback UltraChat-200k OpenbookQA StrategyQA GSM8k MATH MAWPS TabMWP AlpacaEval 2
Benchmarks: ECQA QASC OpenbookQA StrategyQA GSM8k MATH MAWPS TabMWP AlpacaEval 2 Arena-Hard v0.1
Metrics: accuracy length-controlled win rate raw win rate average response length reward margin reward accuracy
Baselines: SFT FIGA IPO ORPO R-DPO SimPO DPO DPO (CW) DPO (EW)
Builds on: DPO IPO R-DPO SimPO FIGA
Closest papers by meaning
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.92)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.92)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.91)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.91)
- MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge (NeurIPS 2026 · similarity 0.90)
- TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.