Research map / Large language models / Alignment and preferences
Discovering Preference Optimization Algorithms with and for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses an LLM (GPT-4) in an iterative loop to propose and score code-level offline preference optimization losses, discovering DiscoPOP (LRML), a sigmoid-blend of logistic and exponential losses that transfers to held-out tasks.
Problem
Offline preference optimization losses (DPO, SLiC, IPO, KTO) are hand-designed by human experts, leaving the large space of possible loss functions under-explored.
Approach
Treat loss design as LLM-driven meta-optimization: GPT-4 proposes Python loss functions in-context from prior losses and their MT-Bench fitness, with unit-test validation; the best loss, DiscoPOP, adaptively blends logistic and exponential losses via a log-ratio-dependent sigmoid weight.
Key results
- DiscoPOP (LRML) raises AlpacaEval 2.0 win rate vs GPT-4 from 11.23% (DPO) to 13.21%, and achieves the best LC win rate vs SFT checkpoint (65.18%).
- Discovery sweep of roughly 100 objectives yields MT-Bench scores up to 7.978 (DBAQL) versus 7.888 for DPO.
- On TL;DR summarization, DPO and PADLL perform best with DiscoPOP slightly below top performers, especially on length-controlled win rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DPO with DBAQL, blending logistic and exponential losses based on loss variability | Zephyr 7B Gemma preference optimization · MT-Bench | score (/ 10): 7.888 -> 7.978 | Helped |
| replace DPO with DiscoPOP, adaptively blending logistic and exponential losses | Zephyr 7B Gemma preference optimization · MT-Bench | score (/ 10): 7.888 -> 7.916 | Helped |
| replace DPO with DiscoPOP | Zephyr 7B Gemma preference optimization · AlpacaEval 2.0 vs GPT-4 | win rate (%): 11.23\pm 0.97 -> 13.21 \pm 1.02 | Helped |
| replace DPO with PADLL | Zephyr 7B Gemma preference optimization · AlpacaEval 2.0 vs GPT-4 | win rate (%): 11.23\pm 0.97 -> 14.07 \pm 1.04 | Helped |
| replace DPO with DiscoPOP | Zephyr 7B Gemma preference optimization · TL;DR summarization vs human preference | win rate (%): 88.27 \pm 1.07 -> 87.63 \pm 1.10 | Hurt |
| use DiscoPOP on IMDb positive-review generation | GPT-2 IMDb preference optimization · IMDb positive sentiment generation | reward and KL-divergence | Mixed |
| train DiscoPOP with β ≤ 0.01 or β ≥ 2.5 | GPT-2 IMDb preference optimization · IMDb positive sentiment generation | training convergence: β ≤ 0.01; β ≥ 2.5 | Hurt |
Surprising findings
- DiscoPOP is non-convex and has negative gradients at rho=0, a property not present in classic convex preference losses.
- MT-Bench evaluation scores were not strongly correlated with training-time validation accuracy; LRML had the worst accuracy despite the best downstream MT-Bench-based fitness.
Limitations
- Relies on closed-source GPT-4 for code generation, limiting reproducibility and increasing cost.
- DiscoPOP fails to converge for beta <= 0.01 or >= 2.5, since beta=0.05 was fixed during discovery.
- Many discovered losses did not respect the beta-scaling of log-ratios, and differences among top losses on AlpacaEval were often not significant.
Experimental setup
Datasets: deita-10k-v0-sft Argilla DPO Mix 7K Reddit TL;DR summarization preferences IMDb preference dataset CIFAR-10
Benchmarks: MT-Bench AlpacaEval 2.0 TL;DR summarization win rate (Alpaca Eval 2.0 library) IMDb reward vs KL
Metrics: MT-Bench score win rate length-controlled win rate reward KL divergence
Baselines: DPO SLiC KTO IPO DPO*
Builds on: DPO SLiC IPO KTO alignment-handbook
Closest papers by meaning
- Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization (NeurIPS 2025 · similarity 0.89)
- Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning (NeurIPS 2025 · similarity 0.88)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.88)
- RePO: Understanding Preference Learning Through ReLU-Based Optimization (NeurIPS 2025 · similarity 0.87)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.87)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.