Research map / Large language models / Alignment and preferences
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Relabels preference pairs with judge reward scores as goal-conditioned prompts (reward-augmented data) and applies DPO, improving instruction-following and academic benchmarks across several 7-9B models.
Problem
Direct preference alignment (DPO) focuses only on relative preferences, ignoring response quality scores from judge models; this causes unlearning of high-quality rejected responses, indiscriminate learning of low-quality chosen ones, and poor generalization to sparse optimal responses.
Approach
Relabel each preference pair twice, conditioning on the goal reward equal to the chosen or rejected response's judge score (via a system prompt like "generate responses of score g"), creating a 2N reward-augmented dataset to which any direct alignment algorithm (DPO) is applied; at inference, condition on the maximum reward g=10.
Key results
- Qwen2-7B-Instruct + DPO (RA) raises AlpacaEval 2.0 LC win rate from 20.93 to 31.17 vs 21.46 for DPO on UltraFeedback.
- SPPO + DPO on reward-augmented data improves LC win rate from 55.60 to 60.97, while DPO on the original UF drops it to 52.75.
- On academic benchmarks, Gemma-2-9B-It average rises from 58.02 to 59.75 with DPO (RA) versus 59.22 with DPO (UF) ... Qwen2-7B average 54.54 vs 53.74.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use reward-augmented rather than original UltraFeedback for DPO | Qwen2-7B-Instruct · Academic benchmarks average | accuracy: 53.74 -> 54.54 | Helped |
| use reward-augmented rather than original UltraFeedback for DPO | Llama-3.1-8B-Instruct · Academic benchmarks average | accuracy: 53.50 -> 54.37 | Helped |
| use reward-augmented rather than original UltraFeedback for DPO | Gemma-2-9B-It · Academic benchmarks average | accuracy: 59.22 -> 59.75 | Helped |
| use reward-augmented rather than original UltraFeedback for DPO | SPPO · Academic benchmarks average | accuracy: 58.77 -> 59.55 | Helped |
| use reward-augmented rather than original UltraFeedback for DPO | Mistral-7B-Instruct-v0.3 · Academic benchmarks average | accuracy: 50.34 -> 50.05 | Hurt |
| add a further DPO round on reward-augmented data (ablation) | SPPO · AlpacaEval 2.0 | LC WR: 55.60 -> 60.97 | Helped |
| add a further DPO round on original UltraFeedback (ablation) | SPPO · AlpacaEval 2.0 | LC WR: 55.60 -> 52.75 | Hurt |
| relabel data with implicit reward from first-round DPO (ablation) | Qwen2-7B-It · AlpacaEval 2.0 | LC WR: 21.46 -> 32.61 | Helped |
| relabel and augment only half of UltraFeedback (ablation) | Qwen2-7B-It · AlpacaEval 2.0 | LC WR: 21.46 -> 29.56 | Helped |
| relabel and augment only half of UltraFeedback (ablation) | Gemma-2-9B-It · AlpacaEval 2.0 | LC WR: 50.70 -> 53.12 | Helped |
| condition on 5-dimensional ArmoRM rewards (ablation) | DPO on Llama-3-8B-Instruct on on-policy data · AlpacaEval 2.0 | LC WR: 53.70 -> 56.57 | Helped |
| condition on goal reward g=8 instead of g=10 at inference (ablation) | Qwen2-7B-It DPO (RA) · AlpacaEval 2.0 | LC WR: 31.17 -> 28.66 | Hurt |
Surprising findings
- Applying DPO on the original UltraFeedback again degrades SPPO, which was already trained on that data, while reward-augmented DPO improves it.
- Relabeling with DPO implicit reward (IRA) can outperform relabeling with GPT-4 judge scores.
Limitations
- Gains on academic benchmarks are mixed; DPO on reward-augmented data sometimes underperforms DPO on UltraFeedback (e.g., Mistral average 50.05 vs 50.34).
- Relies on judge reward scores (GPT-4 or reward model) for relabeling, so binarized data needs implicit-reward relabeling.
- Theoretical guarantees rely on mild assumptions including partial coverage and a reward class with finite covering number.
Experimental setup
Datasets: UltraFeedback
Benchmarks: AlpacaEval 2.0 MT-Bench Arena-Hard-Auto GSM8K GPQA MUSR TruthfulQA BBH ARC Challenge
Metrics: length-controlled win rate win rate MT-Bench score Arena-Hard score accuracy
Baselines: DPO (UltraFeedback) SPPO SimPO OpenChat DPA SteerLM (Info)NCA Reward-aware Preference Optimization
Builds on: DPO UltraFeedback SteerLM Directional Preference Alignment SPPO
Closest papers by meaning
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.94)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.94)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.93)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.93)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.92)
- HPS: Hard Preference Sampling for Human Preference Alignment (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.