Research map / Large language models / Alignment and preferences
MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MPO, a post-processing framework that log-linearly mixes single-objective policies with weights learned by batch stochastic mirror descent, avoiding RL and reward modeling for diverse-preference alignment.
Problem
Aligning LLMs with diverse, heterogeneous human preferences; existing multi-objective RLHF (MORLHF) and MaxMin-RLHF require training multiple reward models and costly, unstable RL runs.
Approach
Shows that the max-min (or linear-scalarized) aggregation of normalized rewards has a closed-form optimal policy equal to a log-linear mixture of single-objective policies, so the mixing weights can be optimized cheaply via batch stochastic mirror descent without RL or reward models.
Key results
- On the Helpful Assistant task with beta=0.5, MPO achieves the highest minimum win rate (53.1%) versus Reward Soups (50.0%) and MaxMin-RLHF (46.1%).
- MPO's computation for weight optimization takes about 2.5 A100 GPU hours versus about 10 A100 GPU hours for PPO-based MORLHF or MaxMin-RLHF.
- In the sentiment/conciseness experiment, the learned weight converges to lambda_1 = 0.386, and MPO outperforms single-reward RLHF on balancing sentiment and conciseness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MPO instead of MaxMin-RLHF | Qwen2.5-7B · Helpful Assistant, β=0.1 | minimum win rate (%): 44.6 -> 46.3 | Helped |
| Use MPO instead of MaxMin-RLHF | Qwen2.5-7B · Helpful Assistant, β=0.5 | minimum win rate (%): 46.1 -> 53.1 | Helped |
| Use MPO instead of Reward Soups with uniform weights | Qwen2.5-7B · Helpful Assistant, β=0.1 | minimum win rate (%): 44.8 -> 46.3 | Helped |
| Use MPO instead of Reward Soups with uniform weights | Qwen2.5-7B · Helpful Assistant, β=0.5 | minimum win rate (%): 50.0 -> 53.1 | Helped |
| Increase β from 0.1 to 0.5 | MPO on Qwen2.5-7B · Helpful Assistant | minimum win rate (%): 46.3 -> 53.1 | Helped |
| Replace MaxMin-RLHF training with MPO weight optimization | Multi-objective alignment · Helpful Assistant | training cost (A100 GPU hours): 10 -> 2.5 | Helped |
| Add the helpful single-objective policy (ablation) | MPO · Helpful Assistant, β=0.5 | helpful reward: -0.11 -> 0.05 | Helped |
| Add the harmless single-objective policy (ablation) | MPO · Helpful Assistant, β=0.5 | harmless reward: -0.02 -> 0.18 | Helped |
| Add the humorous single-objective policy (ablation) | MPO · Helpful Assistant, β=0.5 | humorous reward: -0.10 -> 0.19 | Helped |
Surprising findings
- Under a high KL constraint (beta=0.5), the learned weights are less distinct, and the humorous weight on the harmless-trained policy is near zero at beta=0.1, suggesting the harmless policy substitutes for the humorous one.
Limitations
- MPO requires storing multiple single-objective policies, increasing memory requirements at scale.
- Introducing new preference objectives requires re-optimizing the mixing weights.
- Win-rate evaluation depends on ChatGPT, which is sensitive to prompt design.
Experimental setup
Datasets: IMDb Anthropic/hh-rlhf
Benchmarks: IMDb sentiment and conciseness task HH-RLHF Helpful Assistant (helpful, harmless, humorous)
Metrics: win rate against reference model minimum win rate across objectives Twitter-roBERTa sentiment score response length
Baselines: MaxMin-RLHF MORLHF Reward Soups single-reward RLHF reference model
Builds on: DPO MaxMin-RLHF MORLHF Reward Soups PPO
Closest papers by meaning
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.91)
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.91)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.91)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.91)
- Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.90)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.