Research map / Large language models / Alignment and preferences
Geometric-Averaged Preference Optimization for Soft Preference Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces soft preference labels and weighted geometric averaging of policy likelihoods in DPO-family losses (GDPO, GIPO, GROPO) to improve alignment when preference labels are modestly confident.
Problem
Standard DPO-style alignment assumes binary, deterministic preferences, ignoring distributional human preference and causing over-optimization and objective mismatch, while conservative DPO with linear label smoothing underperforms in text generation.
Approach
Replace winner/loser likelihoods with a soft-label-weighted geometric average of the two responses' likelihoods (exponents p and 1-p), which scales the gradient by (2p-1) so near-equal pairs contribute little gradient; applicable to any DPO-derived objective.
Key results
- GDPO improves binary win rate over DPO on Plasma Plan (85.48% vs 83.16% vs PaLM 2-L) and on Anthropic Harmless vs GPT-4 (43.31% vs 41.02%); average gains of about +2 to +8 points over binary and conservative baselines.
- On Plasma Plan, GIPO and GROPO improve over cIPO and ROPO, with average gains of +3.58% (vs PaLM 2-L binary) and +8.44% (vs GPT-4 binary).
- cDPO achieves best preference classification on average, but DPO drops from 94.0% to 61.6%/66.3% on out-of-distribution GPT-4/GPT-3.5 pairs, which GDPO mitigates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply weighted geometric averaging to DPO, IPO, and ROPO | DPO-family preference optimization · Reddit TL;DR | binary winning rate vs PaLM 2-L: +2.10% | Helped |
| Apply weighted geometric averaging to DPO, IPO, and ROPO | DPO-family preference optimization · Anthropic Helpful | binary winning rate vs PaLM 2-L: +2.90% | Helped |
| Apply weighted geometric averaging to DPO, IPO, and ROPO | DPO-family preference optimization · Anthropic Harmless | binary winning rate vs PaLM 2-L: +4.09% | Helped |
| Apply weighted geometric averaging to DPO, IPO, and ROPO | DPO-family preference optimization · Plasma Plan | binary winning rate vs PaLM 2-L: +3.58% | Helped |
| Apply weighted geometric averaging to DPO, IPO, and ROPO | DPO-family preference optimization · Skewed | binary winning rate vs PaLM 2-L: +5.23% | Helped |
| Apply weighted geometric averaging to DPO, IPO, and ROPO | DPO-family preference optimization · Stairs | binary winning rate vs PaLM 2-L: +4.12% | Helped |
| Use GDPO instead of DPO | PaLM 2-XS preference optimization · Plasma Plan | binary winning rate vs PaLM 2-L: 83.16% -> 85.48% | Helped |
| Use GDPO instead of DPO | PaLM 2-XS preference optimization · Skewed | binary winning rate vs PaLM 2-L: 84.79% -> 86.88% | Helped |
| Use GDPO instead of DPO | PaLM 2-XS preference optimization · Stairs | binary winning rate vs PaLM 2-L: 82.81% -> 84.32% | Helped |
| Use a larger beta for GDPO than DPO (ablation) | GDPO on Plasma Plan · Plasma Plan | binary winning rate | Helped |
Surprising findings
- Conservative DPO (cDPO), which fits the preference distribution better, underperforms in text generation, attributed to objective mismatch with greedy decoding.
- Geometric averaging required a larger beta (0.3 vs 0.1 for DPO) to maintain gradient scale, acting as implicit preference filtering.
Limitations
- Experiments use AI feedback from LLM raters rather than real human majority-vote labels.
- Single seed per setting; statistical tests rely on Wilcoxon over evaluation examples.
- Gains are modest on original RLHF datasets with labels concentrated near 0.5 or 1.0.
Experimental setup
Datasets: Reddit TL;DR Anthropic Helpful Anthropic Harmless Plasma Plan
Benchmarks: Reddit TL;DR Anthropic Helpful Anthropic Harmless Plasma Plan Plasma Plan Skewed Plasma Plan Stairs
Metrics: binary winning rate percentage winning rate preference label classification accuracy Wilcoxon signed-rank test
Baselines: DPO cDPO IPO cIPO ROPO SFT
Builds on: DPO IPO ROPO conservative DPO
Closest papers by meaning
- HPS: Hard Preference Sampling for Human Preference Alignment (ICML 2025 · similarity 0.89)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.89)
- Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.89)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.88)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.88)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.