Research map / Large language models / Alignment and preferences
Distributional Preference Alignment of LLMs via Optimal Transport
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AOT, a distributional preference alignment method that enforces first-order stochastic dominance of chosen over rejected reward distributions via a 1D optimal transport loss solved by sorting, for both unpaired and paired data.
Problem
Existing LLM alignment methods (DPO, KTO, IPO) enforce preferences per sample or via averages, not at the distributional level; unpaired preference data needs a distributional alignment objective.
Approach
Relax first-order stochastic dominance of the positive log-ratio distribution over the negative one into a 1D optimal transport cost with a smooth convex penalty h; the OT problem has a closed-form sorting solution, so the loss is the mean of h over sorted pairs of log-ratios.
Key results
- AOT unpaired reaches 31.3 on AlpacaEval (GPT4) with Merlinite-7B, versus 27.4 for DPO, 24.9 for KTO, and 27.7 for IPO.
- AOT paired scores 29.9 on AlpacaEval (GPT4); AOT unpaired gets 66.2 on Hellaswag and 51.1 on TruthfulQA, the best among the compared methods.
- Logistic loss outperformed least-squares and hinge-squared losses at beta=0.01, and AlpacaEval scores rose with batch size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AOT unpaired instead of the strongest baseline, IPO | Merlinite-7B alignment on UltraFeedback Binarized · AlpacaEval (GPT4) | score: 27.7 -> 31.3 | Helped |
| Use AOT unpaired instead of the strongest baseline, KTO | Merlinite-7B alignment on UltraFeedback Binarized · ARC | score: 82.7 -> 82.5 | Hurt |
| Use AOT unpaired instead of the strongest baseline, DPO | Merlinite-7B alignment on UltraFeedback Binarized · Hellaswag | score: 65.8 -> 66.2 | Helped |
| Use AOT unpaired instead of the strongest baseline, DPO | Merlinite-7B alignment on UltraFeedback Binarized · MMLU | score: 63.1 -> 62.8 | Hurt |
| Use AOT unpaired instead of the strongest baseline, DPO | Merlinite-7B alignment on UltraFeedback Binarized · Truthful | score: 50.6 -> 51.1 | Helped |
| Use AOT unpaired instead of the strongest baseline, KTO | Merlinite-7B alignment on UltraFeedback Binarized · Winogrande | score: 74.9 -> 74.4 | Hurt |
| Use AOT unpaired instead of the strongest baseline, KTO | Merlinite-7B alignment on UltraFeedback Binarized · GSM8K | score: 53.9 -> 51.8 | Hurt |
| Increase batch size (ablation) | AOT · AlpacaEval (Llama3-70B-Instruct judge) | score | Helped |
| Use logistic loss instead of least-squares or squared hinge loss (ablation) | AOT · AlpacaEval | score | Helped |
| Increase beta (ablation) | AOT alignment · LLM benchmark performance | score | Hurt |
Surprising findings
- KTO scores best on GSM8K (53.9) and Winogrande (74.9) while AOT is best on AlpacaEval, so gains are not uniform across benchmarks.
Limitations
- Gains on the Open LLM benchmarks are small or mixed relative to baselines.
- Batch size per GPU limited to 35 samples in the setup, and the theory suggests larger batches help.
- Alignment is limited by the training data and may amplify bias.
Experimental setup
Datasets: UltraFeedback binarized PKU BeaverTails HelpSteer
Benchmarks: AlpacaEval (GPT4) ARC Hellaswag MMLU TruthfulQA Winogrande GSM8K
Metrics: win rate accuracy
Baselines: DPO KTO IPO
Builds on: DPO KTO IPO SLiC
Closest papers by meaning
- Preference Alignment with Flow Matching (NeurIPS 2024 · similarity 0.88)
- Geometric-Averaged Preference Optimization for Soft Preference Labels (NeurIPS 2024 · similarity 0.88)
- Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.88)
- As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.88)
- Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model (ICML 2025 · similarity 0.87)
- No Preference Left Behind: Group Distributional Preference Optimization (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.