Research map / Large language models / Alignment and preferences
Data with High and Consistent Preference Difference Are Better for Reward Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Preference Difference (PD), the reward gap between chosen and rejected responses, as a data filter that derives from an MSE analysis of Bradley-Terry reward-model losses, improving reward-model calibration and RLHF safety win rates.
Problem
Binary preference labels from humans are noisy and inconsistent, so reward model training with the empirical loss is biased relative to the expected loss; it is unclear what properties of preference data make it useful for reward modeling.
Approach
Derive the MSE between empirical and expected BT reward-model losses, show it grows with preference variance and falls with preference intensity, and therefore filter preference pairs by PD = |R(x,y1) - R(x,y2)| from an initial reward model, keeping high-difference pairs; the same idea is extended to DPO via implicit rewards.
Key results
- On Llama-2-7B, PD-filtered reward model (60% data) reached 89.84% overall GPT-4 win rate vs 80.08% for full data and 80.47% for random 40% data.
- On open_llama 3B v2, PD filtering with 40% of data reached 69.92% overall win rate vs 64.84% for full data.
- For DPO on Llama-2-7B, PD filtering with 60% of data achieved 73.63% overall win rate vs 66.82% for full data and 62.21% for random 90%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Filter reward-model training data by retaining the highest-PD samples | OpenLLaMA-3B v2 reward model and RLHF · HH-RLHF test set | overall win rate (%): 64.84 -> 69.92 | Helped |
| Filter reward-model training data by retaining the highest-PD samples | Llama-2-7B reward model and RLHF · HH-RLHF test set | overall win rate (%): 80.08 -> 89.84 | Helped |
| Filter reward-model training data by retaining the highest-PD samples | Mistral-7B reward model and RLHF · HH-RLHF test set | overall win rate (%): 64.26 -> 75.00 | Helped |
| Filter DPO training data by retaining the highest-PD samples | Llama-2-7B DPO · HH-RLHF test set | overall win rate (%): 66.82 -> 73.63 | Helped |
| Use highest-PD filtering instead of random sampling at the same data proportion (ablation) | OpenLLaMA-3B v2 reward model and RLHF · HH-RLHF test set | overall win rate (%): 61.52 -> 69.92 | Helped |
| Use highest-PD filtering instead of random sampling at the same data proportion (ablation) | Llama-2-7B reward model and RLHF · HH-RLHF test set | overall win rate (%): 80.47 -> 89.84 | Helped |
| Use highest-PD filtering instead of random sampling at the same data proportion (ablation) | Mistral-7B reward model and RLHF · HH-RLHF test set | overall win rate (%): 68.75 -> 75.00 | Helped |
| Use highest-PD filtering instead of random sampling (ablation) | Llama-2-7B DPO · HH-RLHF test set | overall win rate (%): 62.21 -> 73.63 | Helped |
| Reduce the retained data proportion below the best range through high-PD filtering (ablation) | Reward models and RLHF · HH-RLHF test set | overall win rate | Hurt |
| Filter data from low to high PD instead of high to low (ablation) | Reward model · HH-RLHF | accuracy: below 50% | Hurt |
| Select higher-PD preference pairs (ablation) | Reward model · HH-RLHF test set | calibrated accuracy | Helped |
| Increase preference difference and consistency in the training data | Reward model · Preference data | MSE between expected and empirical loss | Helped |
Surprising findings
- Filtering from low to high PD produced reward models with accuracy below 50%.
- Using only a fraction of the data (40-60%) outperformed training on the full dataset.
Limitations
- Evaluation relies on GPT-4 as judge with a single safety-oriented dataset (HH-RLHF).
- Filtering requires training an initial reward model to compute PD.
- Theoretical MSE analysis assumes each pair is a binomial sample of the BT probability, which is an idealization of annotator behavior.
Experimental setup
Datasets: HH-RLHF BeaverTails Alpaca Model-Written Evaluation Datasets
Benchmarks: HH-RLHF test set Model-Written Evaluation Datasets (persona, sycophancy, advanced AI risk)
Metrics: GPT-4 win rate calibrated accuracy empirical accuracy matching behavior
Baselines: random data sampling full-data reward model SFT model
Builds on: Bradley-Terry model DPO safe-RLHF PPO
Closest papers by meaning
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.94)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.93)
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.93)
- Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment (ICML 2025 · similarity 0.93)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.92)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.