Research map / Reinforcement learning / RL from feedback
Direct Alignment with Heterogeneous Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes heterogeneous user-type preferences, shows the average user-type reward is the natural alignment target, proves DPO implicitly maximizes Borda count, and gives an approximate first-order DPO correction with paired annotator data plus a consistent loss (with an impossibility on using disagreement data) for full-annotator feedback.
Problem
Standard alignment (RLHF, DPO) assumes a single universal reward, but human preferences are heterogeneous across user types; aligning one policy to a heterogeneous population is unrealizable and aggregation is nontrivial.
Approach
Defines the only well-posed aggregation as an affine combination (average) of user-type rewards, shows DPO implicitly maximizes Borda count, and designs direct alignment losses that use minimal annotator linkage (paired preferences by the same user, enabling variance correction) or full per-type labels (consistent loss using agreement cases), while proving no consistent loss can use disagreement d
Key results
- On semi-synthetic HH-RLHF with three length-based user types, modeling heterogeneity raises agreement with average reward by 4.3 points (about 368 test cases) for both reward learning (93.9% vs 89.6%) and direct alignment (71.7% vs 67.4%).
- On Pew surveys, NBC rankings change under sampling distribution changes in 20% of 1519 questions, and a TV distance of 0.23 suffices to flip the ranking in half the cases.
- The paper proves no consistent M-estimator can estimate the variance term from anonymous data, and that no consistent loss using full per-type labels can use disagreement samples when there are more than three user types.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed consistent loss with maximum annotator information instead of vanilla DPO | Llama-3-8B direct alignment · HH-RLHF test | ordinal agreement with average reward: 67.4% -> 71.7% | Helped |
| Model heterogeneity using the proposed loss with maximum annotator information instead of assuming homogeneity | Llama-3-8B reward learning · HH-RLHF test | ordinal agreement with average reward: 89.6% -> 93.9% | Helped |
| Increase the first-order correction strength alpha (ablation) | Corrected DPO · Synthetic experiments | closeness to optimal policy | Helped |
| Use the consistent loss rather than DPO in the full-information setting | Direct alignment · Synthetic experiments | agreement with optimal policy | Helped |
| Change the sampling distribution of alternatives from uniform | DPO / normalized Borda count · Pew surveys, 1519 questions | frequency of changed NBC rankings: 20% of cases (306 questions) | Hurt |
| Change the sampling distribution of alternatives from uniform | DPO / normalized Borda count · Pew surveys | minimum total variation distance to alter rankings: less than 0.23 | Hurt |
| Use the first-order corrected DPO policy instead of vanilla DPO | DPO · Synthetic experiments | preference for alternatives with δ ∈ {-10, 10}: α=1 | Helped |
Surprising findings
- No sample-efficient consistent direct loss exists under full annotator feedback: any consistent loss must discard samples where user types disagree.
- DPO's induced policy depends on the alternative-response sampling distribution and the preference model temperature even with infinite data, because it maximizes Borda count.
Limitations
- Experiments use simulated annotators and length-based synthetic user types on HH-RLHF rather than real heterogeneous annotators.
- Consistent loss uses only agreement samples, reducing sample efficiency; relies on Bradley-Terry assumptions for the consistent loss.
- Pew analysis estimates rewards via Luce-Shepard model and uses party affiliation as user types, which is a coarse proxy.
Experimental setup
Datasets: HH-RLHF Pew Research Center American Trends Panel surveys
Benchmarks: HH-RLHF (relabeled semi-synthetic) Pew Research surveys (NBC sensitivity)
Metrics: ordinal agreement with average reward raw accuracy minimum TV distance to change NBC ranking
Baselines: DPO reward learning with BT EM-DPO MODPO
Builds on: DPO RLHF Bradley-Terry model IPO
Closest papers by meaning
- Pairwise Calibrated Rewards for Pluralistic Alignment (NeurIPS 2025 · similarity 0.92)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.91)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.90)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.90)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.90)
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.