Research map / Large language models / Alignment and preferences
No Preference Left Behind: Group Distributional Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GDPO, which aligns LLMs to the distribution of preferences within a group by first calibrating a belief distribution and then doing DPO conditioned on sampled beliefs, evaluated on synthetic opinion generation and movie review generation.
Problem
DPO skews toward dominant preferences and fails to capture the distribution of conflicting, pluralistic preferences within a single group, with minority preferences degrading during training.
Approach
Factorize generation into a belief distribution over belief classes followed by belief-conditioned responses; combine a KL belief-calibration loss toward the group's belief statistics with a DPO loss conditioned on the chosen belief, so conflicting preferences are separated by belief.
Key results
- On US opinion generation with Pythia-2.8B, GDPO reaches JSD 0.068 vs 0.352 for DPO and 0.122 for SFT.
- GDPO achieves the best BPC on US opinion generation with GPT-2 Large (0.627 vs 0.441 for DPO and 0.404 for SFT).
- On movie reviews, GDPO attains the lowest JSD and highest BPC among SFT, DPO and GDPO for both GPT-2 Large and Pythia-2.8B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SFT with GDPO | Pythia-2.8B · US, PK, and SA controllable opinion generation | JSD: US: 0.122 -> 0.068; PK: 0.126 -> 0.083; SA: 0.137 -> 0.087 | Helped |
| Replace SFT with GDPO | Pythia-2.8B · US, PK, and SA controllable opinion generation | CBC: US: 0.987 -> 0.989; PK: 0.990 -> 0.991; SA: 0.987 -> 0.990 | Helped |
| Replace the strongest baseline with GDPO | Pythia-2.8B · US, PK, and SA controllable opinion generation | BPC: US: 0.471 -> 0.582; PK: 0.435 -> 0.571; SA: 0.469 -> 0.588 | Helped |
| Replace the strongest baseline with GDPO | Pythia-2.8B · US, PK, and SA controllable opinion generation | RS: US: 0.527 -> 0.554; PK: 0.542 -> 0.582; SA: 0.549 -> 0.536 | Helped |
| Add the belief calibration loss (GDPO first term) (ablation) | GDPO · Controllable opinion generation | JSD | Helped |
| Add belief-conditioned preference alignment loss (GDPO second term) (ablation) | GDPO · Controllable opinion generation | JSD | No clear effect |
| Use GDPO instead of DPO | GPT-2 Large and Pythia-2.8B · Controllable opinion generation training | reward margin | Helped |
| Use GDPO instead of DPO | GPT-2 Large and Pythia-2.8B · Controllable opinion generation training | belief-distribution alignment | Helped |
| Use GDPO instead of DPO | GPT-2 Large · Training efficiency | epoch training time: 13.3 minutes -> 15.48 minutes; 16% more | Hurt |
| Use GDPO instead of SFT and DPO | GPT-2 Large and Pythia-2.8B · Controllable movie review generation | JSD | Helped |
| Use GDPO instead of SFT and DPO | GPT-2 Large and Pythia-2.8B · Controllable movie review generation | BPC | Helped |
Surprising findings
- DPO's reward margin on minority preferences stays negative and deteriorates during training, while its JSD to the target distribution rises early in training.
- Few-shot GPT-4o produces a belief distribution far from the target (JSD about 0.53), worse than fine-tuned small models.
Limitations
- Focuses on a single group; separate models are needed for different groups.
- Requires explicit belief labels present in the data; implicit beliefs need stance detection or latent-variable methods.
Experimental setup
Datasets: GlobalOpinionQA Amazon Movie Review dataset OMDB API metadata HH-RLHF (discussion only)
Benchmarks: GlobalOpinionQA-derived opinion generation (US, Pakistan, South Africa) Amazon movie review generation
Metrics: JSD CBC BPC RS
Baselines: SFT DPO Uniform SFT few-shot prompting In-Context Fine-tuning GPT-4o few-shot
Builds on: Direct Preference Optimization GPO KTO GlobalOpinionQA
Closest papers by meaning
- Group Preference Optimization: Few-Shot Alignment of Large Language Models (ICLR 2024 · similarity 0.92)
- Group Robust Preference Optimization in Reward-free RLHF (NeurIPS 2024 · similarity 0.89)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.89)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.89)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.89)
- Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.