Research map / Large language models / Alignment and preferences
Group Robust Preference Optimization in Reward-free RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRPO, a group-robust reward-free preference optimization method (GR-DPO/GR-IPO) that minimizes worst-case group loss via adaptive multiplicative group weights, with theory for log-linear policies and experiments on synthetic bandit data and GlobalOpinionQA with Gemma-2B.
Problem
LLM alignment with RLHF/DPO-style preference optimization assumes a single preference distribution, so it under-serves minority groups whose labelers have divergent preferences.
Approach
Embed group robustness directly into reward-free DPO/IPO by treating the group-wise losses as a two-player zero-sum game: a max over simplex group weights (updated by multiplicative weights on cumulative loss) against a min over policy parameters, with group information appended to the prompt.
Key results
- On GlobalOpinionQA with Gemma-2B, GR-IPO gives better worst-group test loss and reward accuracy than IPO and reduces the best-worst group loss gap.
- On synthetic bandit experiments with imbalanced groups and different response distributions, GR-DPO/GR-IPO significantly lower worst-case validation loss and reward error versus IS and vanilla methods.
- Theory: a Nash equilibrium exists under log-linear policy parameterization, and the averaged-iterate mirror descent achieves O(T^{-1/2}) error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GR-DPO instead of vanilla DPO | Synthetic group preference data · Synthetic experiments, scenarios (i)–(iii) | max validation loss | Helped |
| Use GR-DPO instead of vanilla DPO | Synthetic group preference data · Synthetic experiments, scenarios (i)–(iii) | max reward error | Helped |
| Use GR-IPO instead of IPO | Synthetic group preference data · Synthetic experiments, scenarios (i)–(iii) | max validation loss | Helped |
| Use GR-IPO instead of IPO | Synthetic group preference data · Synthetic experiments, scenarios (i)–(iii) | max reward error | Helped |
| Use GR-IPO instead of IPO | Gemma-2B · GlobalOpinionQA | maximum group loss | Helped |
| Use GR-IPO instead of IPO | Gemma-2B · GlobalOpinionQA | minimum group reward accuracy | Helped |
| Use GR-IPO instead of IPO | Gemma-2B · GlobalOpinionQA | worst-group log-probability accuracy | Helped |
| Use GR-IPO instead of IPO | Gemma-2B · GlobalOpinionQA | loss imbalance across groups | Helped |
| Use importance sampling instead of vanilla DPO/IPO | Synthetic group preference data with equal group sizes · Synthetic experiments, scenario (ii) | worst-case validation loss | No clear effect |
| Use GRPO when groups are balanced and difficulty levels are comparable | Reward-free RLHF methods · Balanced groups with comparable difficulty | performance | No clear effect |
Surprising findings
- The authors note the max-min objective can be overly conservative and may degrade average performance, motivating a trade-off variant.
- Robustness gives little advantage when groups are balanced and difficulty is comparable.
Limitations
- Max-min objective may be overly conservative and reduce average performance; requires trade-off parameter tuning.
- No advantage when groups are balanced with comparable difficulty.
- Real-data experiments limited to one small model (Gemma-2B) and five countries on a multiple-choice survey dataset.
Experimental setup
Datasets: GlobalOpinionQA synthetic group preference data
Benchmarks: GlobalOpinionQA synthetic bandit preference data
Metrics: max group validation loss max reward error worst-group reward accuracy log-prob accuracy group loss gap
Baselines: DPO IPO IS-DPO IS-IPO
Builds on: direct preference optimization identity preference optimization group distributionally robust optimization Bradley-Terry model nemirovski mirror descent
Closest papers by meaning
- Group Preference Optimization: Few-Shot Alignment of Large Language Models (ICLR 2024 · similarity 0.91)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.90)
- GVPO: Group Variance Policy Optimization for Large Language Model Post-Training (NeurIPS 2025 · similarity 0.90)
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.90)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.90)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.