Research map / Large language models / Alignment and preferences
MallowsPO: Fine-Tune Your LLM with Preference Dispersions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MallowsPO, a DPO generalization based on Mallows ranking models that weights preference learning by a prompt-level dispersion index estimated from the SFT/pretrained model's next-token entropy.
Problem
DPO assumes Bradley-Terry preferences, which cannot characterize varying degrees of human agreement (dispersion) across prompts.
Approach
Replace the Bradley-Terry link with Mallows-theta/phi ranking-model link functions that contain a prompt-dependent dispersion index; the index is approximated cheaply from the entropy of the model's next-token distribution, giving a dispersion-weighted DPO (and MallowsIPO/MallowsSimPO variants).
Key results
- On Llama3-8B-Instruct with beta=0.01, MallowsPO reaches 43.10% LC win rate vs 42.55% for DPO on AlpacaEval 2.
- MallowsSimPO improves SimPO's LC win rate from 50.04% to 51.89%.
- MallowsIPO improves IPO's LC win rate from 43.38% to 44.73%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the MallowsPO dispersion index to DPO | DPO on Llama3-8B-Instruct · Alpaca Eval V2 | LC win rate: 42.55% (0.79) -> 43.10% (0.77) | Helped |
| Add the MallowsPO dispersion index to DPO | DPO on Llama3-8B-Instruct · Alpaca Eval V2 | win rate: 42.02% (1.53) -> 43.02% (1.57) | Helped |
| Add the MallowsPO dispersion index to IPO | IPO on Llama3-8B-Instruct · Alpaca Eval V2 | LC win rate: 43.38% (0.84) -> 44.73% (0.87) | Helped |
| Add the MallowsPO dispersion index to IPO | IPO on Llama3-8B-Instruct · Alpaca Eval V2 | win rate: 43.52% (1.45) -> 44.87% (1.46) | Helped |
| Add the MallowsPO dispersion index to SimPO | SimPO on Llama3-8B-Instruct · Alpaca Eval V2 | LC win rate: 50.04% (0.77) -> 51.89% (0.81) | Helped |
| Add the MallowsPO dispersion index to SimPO | SimPO on Llama3-8B-Instruct · Alpaca Eval V2 | win rate: 42.11% (1.46) -> 43.76% (1.47) | Helped |
| Use MallowsPO-φ instead of DPO | GPT-2-large · IMDB conditional generation | KL divergence at the same reward-model accuracy | Helped |
| Use MallowsPO-θ instead of DPO | GPT-2-large · IMDB conditional generation | reward-model accuracy and KL tradeoff | No clear effect |
| Use MallowsPO instead of DPO | Pythia-2.8B · Anthropic HH test split | GPT-4 preference win rate | Helped |
| Use MallowsPO instead of DPO | Pythia-2.8B · SE and SHP out-of-distribution evaluation | GPT-4 preference win rate | Helped |
| Use MallowsPO-θ instead of DPO | Pythia-2.8B · Anthropic HH, SE, and SHP | ArmoRM win rate | Helped |
| Use MallowsPO-φ instead of DPO | Synthetic five-arm bandit · Synthetic bandit | reward at a fixed KL divergence | Helped |
Surprising findings
- Reported MallowsPO gains on AlpacaEval 2 are small (under 2 points LC win rate), with overlapping standard deviations across seeds.
- Controllable IMDB generation shows little dispersion, so MallowsPO-theta performs close to DPO there.
Limitations
- The entropy-based dispersion proxy is heuristic and only validated on a synthetic bandit setup.
- Choice of beta and the scaling constant phi* lacks guidelines per the authors.
- Gains on the Llama3 benchmark are modest relative to seed variance.
Experimental setup
Datasets: IMDB Anthropic HH UltraFeedback
Benchmarks: IMDB controllable generation Anthropic HH H4 Stack Exchange preferences Stanford Human Preferences AlpacaEval 2
Metrics: win rate LC win rate reward model score KL divergence
Baselines: DPO IPO SimPO KTO f-DPO ODPO
Builds on: DPO Bradley-Terry model Mallows ranking model PsiPO SimPO
Closest papers by meaning
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.91)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
- SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024 · similarity 0.91)
- Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.91)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.90)
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.