Research map / Learning theory and algorithms / Prediction and decision losses
Generalized Preference Optimization: A Unified Approach to Offline Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Generalized Preference Optimization (GPO), a family of offline preference losses parameterized by a convex function f that unifies DPO, IPO and SLiC, and analyzes the regularization they induce versus KL.
Problem
Offline preference optimization methods (DPO, IPO, SLiC) are proposed separately with different losses, and it is unclear how their regularization against the reference policy relates to the KL-regularized RLHF objective.
Approach
Views reward learning as supervised binary classification with a convex surrogate loss f applied to the log-ratio difference, so any convex classification loss yields an offline preference objective; analyzes the effective regularization via Taylor expansion and compares the offline squared-loss regularizer with the KL divergence.
Key results
- Squared and truncated squared losses peak at beta around 1, while other GPO variants peak at beta around 10, indicating different natural regularization strengths.
- Offline squared loss tracks KL divergence only near the reference policy; KL can vary by an order of magnitude more than the offline regularizer, hinting at instability when enforcing KL offline.
- In the summarization side-by-side evaluation, GPO variants perform mostly on par, with best win rates at beta in [0.1, 1].
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use different convex loss variants (DPO, IPO, SLiC, and other GPO losses), tuning β for each | Offline preference optimization on the summarization task · Summarization | win rate | No clear effect |
| Increase β beyond the best-performing range | GPO variants on the summarization task · Summarization | win rate | Hurt |
| Set β in the range [0.1,1] | GPO variants on the summarization task · Summarization | win rate | Helped |
| Increase β from about 1 toward 100 for the logistic loss | DPO · Summarization, golden-policy evaluation | golden win rate | Hurt |
| Use β=0.01 instead of a more suitable β for the logistic loss | DPO · Summarization, golden-policy evaluation | golden win rate | Hurt |
| Use squared or truncated squared loss rather than the other tested GPO losses at large β | GPO variants · Summarization, golden-policy evaluation | KL divergence | Helped |
| Use logistic loss rather than the other compared GPO variants | GPO variants · Summarization, KL versus μ-weighted squared loss analysis | KL divergence variation | Hurt |
| Increase β for squared or truncated squared loss relative to other GPO variants | GPO variants · Summarization, golden-policy evaluation | golden win rate | Mixed |
| Continue training beyond the point of peak golden performance | GPO variants · Summarization, golden-policy evaluation | golden win rate | Hurt |
| Use logistic, exponential, or Savage loss rather than hinge, smoothed quadratic, or squared loss | GPO policies in the 3-action bandit · 3-action bandit | distance of πθ(y1) from πref(y1) | Hurt |
| Use hinge, smoothed quadratic, or squared loss rather than logistic, exponential, or Savage loss | GPO policies in the 3-action bandit · 3-action bandit | distance of πθ(y1) from πref(y1) | Helped |
| Minimize the offline μ-weighted squared loss instead of directly minimizing KL divergence | Offline policy optimization · Mixture of Gaussians counterexample | KL divergence | Mixed |
Surprising findings
- Offline preference losses with fast-decaying tails (logistic, exponential, Savage) push the policy to a greedy action in the 3-action bandit, even with regularization, while hinge and squared variants preserve closer distance to the reference.
- The offline squared regularizer is not a reliable proxy for KL: minimizing it may not decrease KL globally, as shown by a mixture-of-Gaussians counterexample.
Limitations
- Theory relies on a reward-maximization (Bradley-Terry-like) view that may fail for complex non-transitive preferences.
- Taylor-expansion analysis is only valid near the reference policy and does not cover non-smooth losses like hinge (SLiC).
- Framework only covers contrastive-form offline losses, not supervised-learning-based losses.
Experimental setup
Datasets: open source summarization dataset with human feedback labels (Reddit TL;DR-style)
Benchmarks: summarization side-by-side win rate (2000 examples) synthetic golden preference win rate
Metrics: win rate KL divergence 90th percentile win rate
Baselines: DPO IPO SLiC online IPO
Builds on: DPO IPO SLiC
Closest papers by meaning
- As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.88)
- Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.88)
- Meta-Learning Objectives for Preference Optimization (NeurIPS 2025 · similarity 0.88)
- Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences (NeurIPS 2026 · similarity 0.88)
- The Importance of Online Data: Understanding Preference Fine-tuning via Coverage (NeurIPS 2024 · similarity 0.87)
- Distributional Preference Alignment of LLMs via Optimal Transport (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.