Research map / Large language models / Alignment and preferences
Multi-Reference Preference Optimization for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MRPO, a closed-form direct preference optimization that uses multiple reference LLMs via a virtual aggregated reference policy, stabilized by clipped trust regions and adaptive reference weights.
Problem
Direct preference optimization (DPO) constrains the tuned LLM to a single reference model, ignoring the knowledge available in multiple pretrained/SFT LLMs; naive multi-reference combinations are unstable and suboptimal.
Approach
Derives a closed-form optimum for a surrogate lower bound of the multi-reference KL-regularized RLHF objective, yielding a DPO-style loss with a virtual reference policy that harmonically aggregates K reference policies; adds clipped trust-region optimization (CTRO) to keep non-initial references close to the initial one, and adaptive reference weighting (ARWC) based on each reference's discrimina
Key results
- On small preference datasets, MRPO improves over DPO by roughly 3-7% (L←M) and 1-4% (M←L) in test preference accuracy.
- On HelpSteer, Ultrafeedback and Nectar, MRPO raises reward margin by 10-20% over DPO and accuracy by about 1-5%.
- On Open LLM Leaderboard tasks with K=3 references, MRPO averages 1.1%, 1.0% and 1.3% above DPO across three base models; TinyLlama distillation gains are modest (39.4 vs 39.2 DPO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multiple reference models with MRPO instead of DPO | Llama and Mistral preference tuning · HelpSteer, Ultrafeedback, and Nectar; L←M and M←L | preference accuracy: 68.9{ $\pm$0.4} -> 73.6 $\pm$1.6; 70.8{ $\pm$4.3} -> 71.6 $\pm$5.2; 69.8{ $\pm$0.9} -> 72.9 $\pm$2.9; 72.0{ $\pm$1.1} -> | Helped |
| Use three reference models with MRPO instead of DPO | Open LLM Leaderboard GLU evaluation · Open LLM Leaderboard average across Mistral, Llama, and Qwen initializations | GLU score: 65.0 -> 66.1; 51.4 -> 52.4; 54.0 -> 55.3 | Helped |
| Use MRPO with Mistral as an additional reference | TinyLlama-1.1B · Open LLM Leaderboard average | GLU score: 39.2 -> 39.4 | Helped |
| Use MRPO with three references rather than DPO | Mistral base model · GSM8K | score: 53.7 -> 60.5 | Helped |
| Use MRPO with three references rather than DPO | Llama base model · TruthfulQA | score: 39.5 -> 45.3 | Helped |
| Use MRPO with three references rather than DPO | Qwen base model · GSM8K | score: 18.7 -> 23.7 | Helped |
| Set εmax to 0.1 instead of 0 (DPO) (ablation) | MRPO with diverse reference families · S3, setting (ii) | accuracy: 0.54 -> 0.61 | Helped |
| Set εmax to 10^6 (no clipping) rather than 0 (DPO) (ablation) | MRPO with different-family references · S3, setting (ii) | accuracy: 0.54 -> 0.45 | Hurt |
| Set εmax to 1 rather than 0 (DPO) (ablation) | MRPO with different-family references · S3, setting (ii) | accuracy: 0.54 -> 0.49 | Hurt |
| Set εmax to 0.01 rather than 0 (DPO) (ablation) | MRPO with different-family references · S3, setting (ii) | accuracy: 0.54 -> 0.55 | No clear effect |
| Use adaptive ε rather than fixed ε=0.1 (ablation) | MRPO with Llama and Mistral references · S1, S2, and S3 | preference accuracy | Helped |
| Use adaptive α rather than fixed α values (ablation) | MRPO with Llama and Mistral references · S1, S2, and S3 | preference accuracy | Mixed |
Surprising findings
- Naive Multi-DPO often underperforms standard DPO, even with the proposed clipping and adaptive weighting.
- Gains on TinyLlama (weak base model) are small, only 0.2% over DPO.
Limitations
- Limited to up to three reference models of modest size (7B and smaller).
- Limited improvement on small base models such as TinyLlama.
- Gains on some benchmarks are small and within run-to-run variance.
Experimental setup
Datasets: comparison_gpt4_data_en Neural-DPO distilabel-math-preference-dpo HelpSteer Ultrafeedback Nectar
Benchmarks: GSM8K TruthfulQA HellaSwag MMLU Arc-easy Winogrande
Metrics: preference accuracy reward margin average accuracy on Open LLM Leaderboard tasks
Baselines: DPO Multi-DPO SFT
Builds on: DPO RLHF Multi-DPO
Closest papers by meaning
- Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.93)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.92)
- Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization (NeurIPS 2025 · similarity 0.92)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.92)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)
- TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.