Research map / Large language models / Alignment and preferences

MaxMin-RLHF: Alignment with Diverse Human Preferences

ICML 2024 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that single-reward RLHF cannot align with diverse, minority-skewed human preferences, and proposes MaxMin-RLHF, which learns a mixture of reward models via EM and optimizes the worst-off subgroup's reward (Egalitarian max-min objective).

Problem

Single-reward RLHF averages over heterogeneous human subpopulations, so minority preferences are under-served; the paper asks whether a single reward pipeline can align with diverse preferences.

Approach

Derives an impossibility/alignment-gap lower bound for single-reward RLHF under preference diversity, then learns a mixture of reward models with EM and trains a policy to maximize the minimum subgroup utility (Egalitarian MaxMin objective) using PPO.

Key results

• On Tulu2-7B with P3 (GPT-4 judge), MaxMin achieves 56.67% average win rate vs 54.24% for the 1:1 single-reward mixture and 50.50% for 10:1.
• Single reward model test accuracy on minority group drops from 0.704 at 1:1 ratio to 0.420 at 10:1 ratio.
• MaxMin on P1 gets 60.00% on P1B vs 45.08% for 10:1 single-reward baseline.

Measured effects reported in the paper

Surprising findings

• Single-reward RLHF on the 2:1 to 10:1 imbalanced data improves majority accuracy but degrades minority accuracy, with total accuracy falling as the imbalance grows.

Limitations

• Assumes a known or fixed number of user clusters (|U|=2 in experiments).
• Evaluation relies on simulated users and GPT-4 judgments rather than real human annotators.
• The impossibility result depends on linear reward parametrization and bounded features.

Experimental setup

Datasets: IMDb GPT4-Alpaca Koala evaluation set

Benchmarks: Koala evaluation prompts (50 instances) IMDb sentiment/conciseness

Metrics: pairwise win rate vs base model reward model test accuracy sentiment score conciseness

Baselines: single reward RLHF 1:1 ratio mixture 2:1 ratio mixture 6:1 ratio mixture 10:1 ratio mixture

Builds on: RLHF Bradley-Terry model PPO DPO

Closest papers by meaning

• Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.93)
• Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.92)
• Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.92)
• Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.92)
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.92)
• MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment (ICML 2025 · similarity 0.91)