Research map / Large language models / Alignment and preferences
Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes alignment potential M_AP, the gap between a preference pair's explicit (reward-model) margin and the current model's implicit margin, to select high-quality preference data for DPO/SimPO and for self-play (eva-style) data generation.
Problem
Existing data quality metrics based on explicit reward margins or implicit reward margins give contradictory rankings for the same preference pairs, and explicit margins are vulnerable to reward model noise.
Approach
At the aligned optimum, implicit and explicit reward margins coincide, so the gap between the target explicit margin and the current implicit margin measures how much a pair can still improve alignment; a regularized (absolute-value) form avoids rewarding reward-model noise. The metric is also used in an evolve-then-select self-play pipeline.
Key results
- Training on top-40% SimPO data selected by M_AP outperforms uniform sampling by 2.5 and 6.2 points on AlpacaEval LC for Llama and Gemma (per paper text).
- In the self-play setting, +ours-10k with SimPO reaches 34.6 Arena-Hard WR on Llama-3-8B-Instruct vs 32.6 for +eva-10k.
- A 30k subset selected by M_AP outperforms the full UltraFeedback dataset, improving 2.3 Alpaca LC and 3.4 Arena Hard points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select additional evolved preference data with alignment potential instead of eva | DPO on Llama-3-8b-instruct · Arena Hard | win rate: 35.0 -> 35.4 | Helped |
| Select additional evolved preference data with alignment potential instead of eva | DPO on Llama-3-8b-instruct · AlpacaEval 2 | length-controlled win rate: 39.97 -> 42.82 | Helped |
| Select additional evolved preference data with alignment potential instead of eva | DPO on Llama-3-8b-instruct · AlpacaEval 2 | win rate: 40.74 -> 45.00 | Helped |
| Select additional evolved preference data with alignment potential instead of eva | DPO on Gemma-2-9b-it · Arena Hard | win rate: 56.0 -> 56.6 | Helped |
| Select additional evolved preference data with alignment potential instead of eva | DPO on Gemma-2-9b-it · AlpacaEval 2 | length-controlled win rate: 63.32 -> 63.88 | Helped |
| Select additional evolved preference data with alignment potential instead of eva | DPO on Gemma-2-9b-it · AlpacaEval 2 | win rate: 62.10 -> 64.52 | Helped |
| Select additional evolved preference data with alignment potential instead of eva | SimPO on Llama-3-8b-instruct · Arena Hard | win rate: 32.6 -> 34.6 | Helped |
| Select additional evolved preference data with alignment potential instead of eva | SimPO on Llama-3-8b-instruct · AlpacaEval 2 | length-controlled win rate: 41.12 -> 43.76 | Helped |
| Select additional evolved preference data with alignment potential instead of eva | SimPO on Llama-3-8b-instruct · AlpacaEval 2 | win rate: 39.94 -> 42.00 | Helped |
| Select additional evolved preference data with alignment potential instead of eva | SimPO on Gemma-2-9b-it · Arena Hard | win rate: 61.0 -> 60.9 | Hurt |
| Select additional evolved preference data with alignment potential instead of eva | SimPO on Gemma-2-9b-it · AlpacaEval 2 | length-controlled win rate: 66.78 -> 66.85 | Helped |
| Select additional evolved preference data with alignment potential instead of eva | SimPO on Gemma-2-9b-it · AlpacaEval 2 | win rate: 66.86 -> 68.34 | Helped |
Surprising findings
- Explicit and implicit reward margin rankings can contradict each other on the same preference pairs, and the reward model can be noisy enough that large explicit margins mislabel data.
- Using the absolute-value (regularized) form of the gap improves agreement with GPT-4 over the unregularized one-sided M_+ metric.
Limitations
- Requires tuning parameter beta (or alpha) to balance explicit and implicit margins.
- Evaluated only with DPO and SimPO objectives.
Experimental setup
Datasets: UltraFeedback SimPO llama3-ultrafeedback-armorm SimPO gemma2-ultrafeedback-armorm SimPO llama3-ultrafeedback
Benchmarks: AlpacaEval 2 Arena-Hard
Metrics: win rate length-controlled win rate
Baselines: explicit reward margin selection implicit reward margin selection uniform sampling eva UltraFeedback uf-10k
Builds on: DPO SimPO eva EvolInstruct
Closest papers by meaning
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.94)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.93)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.93)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.92)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.92)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.