Research map / Large language models / Alignment and preferences
DPO design choices
Study data, scheduling, dynamic beta, and multi-reference variants of direct preference optimization.
73 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2024–2026) · an approach in Alignment and preferences, keyword cluster “preference optimization · dpo · direct preference”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
6 papers in 2023–24 and 67 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 4.6% to 12.8%.
Open questions the data points to
- Papers report DPO on AlpacaEval 2 (length-controlled win rate (%)) anywhere from 18.2 to 36.3: comparisons against it depend on whose number you take.
- Papers report DPO on Arena-Hard (win rate (%)) anywhere from 10.4 to 33.6: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2024 · SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024 · 39 citations)
- 2025 · Robust Multi-Objective Preference Alignment with Online DPO (AAAI 2025 · 2 citations)
- 2026 · SMPRO: Self-Supervised Visual Preference Alignment via Differentiable Multi-Preference Multi-Group Ranking (AAAI 2026 · 1 citation)
Earliest papers here
- SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024 · 39 citations)
- $\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$ (NeurIPS 2024 · 11 citations)
- Iterative Reasoning Preference Optimization (NeurIPS 2024 · 9 citations)
What has been tried
35 papers here report 405 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 8%, hurt 21%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| remove length normalization | AlpacaEval 2 · length-controlled win rate (%) | 21.5 -> 11.9 | Hurt | SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024 · 39 citations) |
| Add dynamic β without data filtering | Anthropic HH, Pythia-410M and Pythia-2.8B · win rate | 26.19 -> 27.15; 51.51 -> 55.19 | Helped | $\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$ (NeurIPS 2024 · 11 citations) |
| Add an NLL term to standard DPO | GSM8K test · accuracy | 61.8% to 73.1% | Helped | Iterative Reasoning Preference Optimization (NeurIPS 2024 · 9 citations) |
| Increase the reference-model influence with larger γ in γ-scaled DPO | Win rate and ranking accuracy during alignment · correlation between ranking accuracy and win rate | Helped | Preference Learning Algorithms Do Not Learn Preference Rankings (NeurIPS 2024 · 5 citations) | |
| Use Dirichlet alpha=1.0 (higher) instead of a low alpha for objective weight sampling | Anthropic-HH · steerability of Pareto front | Helped | Robust Multi-Objective Preference Alignment with Online DPO (AAAI 2025 · 2 citations) | |
| add Adaptive Step Reward | MATH · pass@1 accuracy | 55.08 -> 55.46 | Helped | TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees (ICLR 2025 · 2 citations) |
| Set εmax to 0.1 instead of 0 (DPO) | S3, setting (ii) · accuracy | 0.54 -> 0.61 | Helped | Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · 1 citation) |
| use distance-based rather than random negative-element sampling | Mind2Web overall · SSR | 61.1 -> 63.5 | Helped | WEPO: Web Element Preference Optimization for LLM-based Web Navigation (AAAI 2025 · 1 citation) |
| train only the language model and freeze the diffusion model | LLM + diffusion image sequences · OC Ratio | 32% -> 65% | Helped | Aligning Compound AI Systems via System-level DPO (NeurIPS 2025 · 1 citation) |
| Add high-level semantic reasoning to SFT | Ablation study · Usability Rate | 28% -> 33% | Helped | OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization (NeurIPS 2025 · 1 citation) |
| Add data augmentation with harmful-response prefixes | Prefilling · ASR | Helped | Improving LLM Safety Alignment with Dual-Objective Optimization (ICML 2025 · 0 citations) | |
| Increase the discount factor γ | Synthetic preference-drift experiment · training steps to final test accuracy | Helped | Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift (ICML 2025 · 0 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| DPO | AlpacaEval 2 | length-controlled win rate (%) | 18.2 – 36.3 | 3 papers, e.g. NeurIPS 2024 and ICLR 2025 |
| DPO | Arena-Hard | win rate (%) | 10.4 – 33.6 | 2 papers, e.g. NeurIPS 2024 and ICLR 2025 |
| SimPO | AlpacaEval 2 | length-controlled win rate | 21.3 – 28.5 | 2 papers, e.g. ICLR 2025 and NeurIPS 2024 |
What reviewers will expect
Datasets and baselines named in the 41 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- UltraFeedback · 18 of 41 papers
- MATH · 6 of 41 papers
- Anthropic-HH · 5 of 41 papers
- GSM8K · 3 of 41 papers
- AlpacaEval 2 · 3 of 41 papers
- SORRY-Bench · 2 of 41 papers
- MMLU · 2 of 41 papers
- HellaSwag · 2 of 41 papers
Baselines
- DPO · 22 of 41 papers
- IPO · 15 of 41 papers
- KTO · 11 of 41 papers
- SimPO · 9 of 41 papers
- ORPO · 7 of 41 papers
- SFT · 6 of 41 papers
- CPO · 6 of 41 papers
- R-DPO · 5 of 41 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 73 papers; from OpenAlex and the papers' affiliations).
- Google · 4 papers
- Google DeepMind · 4 papers
- New York University · 3 papers
- Rutgers University · 2 papers
- University of Virginia · 2 papers
- Princeton University · 2 papers
- Genentech · 2 papers
- University of Science and Technology of China · 2 papers
- ByteDance · 2 papers
- Tsinghua University · 2 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Robust and calibrated DPO (118 papers)
- Active and token-level preference learning (58 papers)
Elsewhere in the topic
- Steering and correcting LLM behavior (127 papers)
- LLM-as-judge and feedback data (111 papers)
- Reward modeling for diverse preferences (90 papers)
- Reward model design and preference models (88 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).