AI Paper Atlas

Research map / Large language models / Alignment and preferences

Is On-Policy Data always the Best Choice for Direct Preference Optimization-Based LM Alignment?

Zetian Sun, Dongfang Li, Xuhui Chen, Baotian Hu, Min Zhang

ICLR 2026 · Large language models › Alignment and preferences · approach: DPO design choices

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use on-policy instead of off-policy preference candidates for iteration 1 (ablation)DPO · AlpacaEval 2.0LC win rate: 27.73 -> 34.04Improved
use on-policy instead of off-policy preference candidates for iteration 1 (ablation)DPO · AlpacaEval 2.0LC win rate: 20.77 -> 13.70Hurt
use on-policy instead of off-policy preference candidates for iteration 1 (ablation)DPO · AlpacaEval 2.0LC win rate: 5.97 -> 4.21Hurt
use on-policy candidates in iteration 2 after off-policy iteration 1 (ablation)DPO · AlpacaEval 2.0LC win rate: Zephyr 23.77 -> 33.28Improved
train second iteration on diverse off-policy data again (PC_off then PC_off) (ablation)DPO · AlpacaEval 2.0LC win rate: Llama-3 27.73 -> 27.83No clear effect
use high-quality off-policy candidates from Llama-3 (PC_llama) instead of PC_off at second iteration (ablation)DPO on Zephyr · AlpacaEval 2.0LC win rate: 23.77 -> 29.32Improved
use high-quality PC_llama instead of diverse PC_off at first iteration (ablation)DPO on Zephyr · AlpacaEval 2.0LC win rate: 20.77 -> 13.53Hurt
choose candidate source with the boundary measurement algorithmDPO · AlpacaEval 2.0, Llama-3 and Phi-2stage prediction: Llama-3 scores 0.62/0.66/0.69; Phi-2 scores 0.23/0.25/0.23Improved
boundary score for Zephyr trained with PC_on firstboundary measurement algorithm · AlpacaEval 2.0, Zephyrstage prediction: score 0.58 but PC_off still better (+8.5 vs +5.5)Mixed
apply the stage conclusions to Qwen2.5-1.5B and Pythia-6.9BDPO · AlpacaEval 2.0LC win rate: PC_off beats PC_on given same checkpoint, injection stageImproved
replace DPO with SLiC-HF loss (ablation)Llama-3 off->on · AlpacaEval 2.0LC win rate: 28.18 -> 12.66Hurt
run boundary measurement on 2,000 promptsboundary measurement algorithm · UltraFeedback test_prefscompute overhead: 3.2% of a single training epochImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy