Research map / Large language models / Alignment and preferences
Is On-Policy Data always the Best Choice for Direct Preference Optimization-Based LM Alignment?
Zetian Sun, Dongfang Li, Xuhui Chen, Baotian Hu, Min Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use on-policy instead of off-policy preference candidates for iteration 1 (ablation) | DPO · AlpacaEval 2.0 | LC win rate: 27.73 -> 34.04 | Improved |
| use on-policy instead of off-policy preference candidates for iteration 1 (ablation) | DPO · AlpacaEval 2.0 | LC win rate: 20.77 -> 13.70 | Hurt |
| use on-policy instead of off-policy preference candidates for iteration 1 (ablation) | DPO · AlpacaEval 2.0 | LC win rate: 5.97 -> 4.21 | Hurt |
| use on-policy candidates in iteration 2 after off-policy iteration 1 (ablation) | DPO · AlpacaEval 2.0 | LC win rate: Zephyr 23.77 -> 33.28 | Improved |
| train second iteration on diverse off-policy data again (PC_off then PC_off) (ablation) | DPO · AlpacaEval 2.0 | LC win rate: Llama-3 27.73 -> 27.83 | No clear effect |
| use high-quality off-policy candidates from Llama-3 (PC_llama) instead of PC_off at second iteration (ablation) | DPO on Zephyr · AlpacaEval 2.0 | LC win rate: 23.77 -> 29.32 | Improved |
| use high-quality PC_llama instead of diverse PC_off at first iteration (ablation) | DPO on Zephyr · AlpacaEval 2.0 | LC win rate: 20.77 -> 13.53 | Hurt |
| choose candidate source with the boundary measurement algorithm | DPO · AlpacaEval 2.0, Llama-3 and Phi-2 | stage prediction: Llama-3 scores 0.62/0.66/0.69; Phi-2 scores 0.23/0.25/0.23 | Improved |
| boundary score for Zephyr trained with PC_on first | boundary measurement algorithm · AlpacaEval 2.0, Zephyr | stage prediction: score 0.58 but PC_off still better (+8.5 vs +5.5) | Mixed |
| apply the stage conclusions to Qwen2.5-1.5B and Pythia-6.9B | DPO · AlpacaEval 2.0 | LC win rate: PC_off beats PC_on given same checkpoint, injection stage | Improved |
| replace DPO with SLiC-HF loss (ablation) | Llama-3 off->on · AlpacaEval 2.0 | LC win rate: 28.18 -> 12.66 | Hurt |
| run boundary measurement on 2,000 prompts | boundary measurement algorithm · UltraFeedback test_prefs | compute overhead: 3.2% of a single training epoch | Improved |
Closest papers by meaning
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.93)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.92)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.92)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.92)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
