Research map / Large language models / Alignment and preferences

3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

ICLR 2025 · Large language models › Alignment and preferences · approach: DPO design choices · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes DPO's failure modes via gradient analysis, names them the 3D-properties (drastic rejected-likelihood drop, response suppression, dispersion to unseen responses), and proposes Flex-DPO and SFT-regularized DPO, validated on a toy model and Baichuan2 math/instruction-following tasks.

Problem

DPO is cheaper than PPO but underperforms RM-based alignment in state-of-the-art LLMs; its training dynamics (falling likelihood of both responses, mass leaking to unseen outputs, instability) are poorly explained theoretically.

Approach

Gradient analysis shows the rejected-response gradient explodes as its likelihood approaches zero while the chosen-response gradient vanishes, producing three named properties; mitigations are per-response beta (Flex-DPO) and an added SFT loss on chosen responses.

Key results

• DPO in Scenario 1 (on-policy) raised Baichuan2-13B 5-point accuracy on MATH* from 32.237% to 37.132%, the best of the four data scenarios.
• On poem generation, PPO outperformed DPO on most metrics (e.g., Tone Pattern 0.82 vs 0.75, Title 1 vs 0.78), and DPO was lower on slogan content (0.70 vs 0.78 for PPO).
• Flex-DPO was reported to consistently outperform vanilla DPO, IPO and SLiC on MATH* and SuperCLUE-Math.

Measured effects reported in the paper

Surprising findings

• Off-policy chosen responses from the dataset (Scenario 2) degraded DPO relative to on-policy data, and in Baichuan2-13B Scenario 2 and 3 underperformed the base model on the 4&5 metric.
• Several DPO variants (SLiC, IPO) underperformed vanilla DPO on instruction-following poem generation despite helping on math.

Limitations

• Over-reducing the rejected-response beta causes regression toward SFT behavior, so the regularization has a narrow useful range.
• Several in-house datasets (poem, slogan) are not released, limiting reproducibility; the toy model is an abstract simplification of real LLM training.

Experimental setup

Datasets: MATH SuperCLUE-Math HH-RLHF UltraFeedback in-house poem dataset in-house slogan dataset

Benchmarks: MATH* SuperCLUE-Math poem generation (in-house) slogan generation (in-house) HH-RLHF eval

Metrics: accuracy on 5-point and 4&5-point GPT-4 scores rule-based poem and slogan constraint scores RM/DPO preference accuracy

Baselines: PPO IPO SLiC base model RM training

Builds on: DPO PPO IPO SLiC SimPO

Closest papers by meaning

• What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.94)
• Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.93)
• Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.93)
• Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.92)
• Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.92)
• Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)