Research map / Large language models / Alignment and preferences
Less is More: Improving LLM Alignment via Preference Data Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BeeS, a Bayesian-aggregation data selection method for DPO that combines external reward margins and implicit DPO margins to pick high-margin preference pairs, reaching full-dataset-level or better AlpacaEval 2.0 performance with about 10% of the data.
Problem
Noisy preference labels from imperfect reward sources cause parameter shrinkage in DPO, and training on the full preference dataset is costly and can hurt performance; the paper asks which preference data to keep for DPO.
Approach
Theory suggests exogenous label noise shrinks learned parameters and that selecting high-margin pairs counteracts this. BeeS aggregates external-reward and implicit-DPO-reward margins into one preference probability via a Bayesian product formula, deprioritizing any pair with a low margin from any single source.
Key results
- On AlpacaEval 2.0 with 6k samples (10% of UltraFeedback), BeeS gets 30.46% raw WR on Llama-3-8B-Instruct vs 26.54% for full-set DPO.
- Across four Llama-3-8B settings, BeeS subsets outperform full-set training by roughly 4 points.
- Iterative DPO with BeeS using 25% of online data reaches 48.49% win rate and 54.99% LC win rate on AlpacaEval 2.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select 2,000 samples with BeeS rather than train on the full dataset | DPO on Llama-3.2-3B · TL;DR | win rate: 36.75 -> 83.25 | Helped |
| Select 2,000 samples with BeeS rather than train on the full dataset | DPO on Llama-3.2-3B · HH | win rate: 92.00 -> 90.25 | No clear effect |
| Select 2,000 samples with BeeS rather than train on the full dataset | DPO on Llama-3.2-3B · UltraFeedback | win rate: 80.99 -> 91.68 | Helped |
| Select 6,000 samples with BeeS rather than train on the full dataset | DPO on Llama-3-Base (8B) · AlpacaEval 2.0, UltraFeedback | WR (%): 15.30 -> 19.09 | Helped |
| Select 6,000 samples with BeeS rather than train on the full dataset | DPO on Llama-3-Instruct (8B) · AlpacaEval 2.0, UltraFeedback | WR (%): 26.54 -> 30.46 | Helped |
| Select 6,000 samples with BeeS rather than train on the full dataset | DPO on Llama-3-Base (8B) · AlpacaEval 2.0, Llama-UltraFeedback | WR (%): 16.45 -> 20.01 | Helped |
| Select 6,000 samples with BeeS rather than train on the full dataset | DPO on Llama-3-Instruct (8B) · AlpacaEval 2.0, Llama-UltraFeedback | WR (%): 32.44 -> 36.47 | Helped |
| Use implicit-margin-N selection instead of random selection | DPO on Llama-3.2-3B · HH | win rate: 84.25 -> 32.00 | Hurt |
| Use a 3k-sample BeeS subset rather than the full dataset (ablation) | DPO on Llama-3-Instruct (8B), Llama-UltraFeedback, β=0.01 · AlpacaEval 2.0 | win rate | No clear effect |
| Reduce β to 0.01 for DPO training on Ex/Im-P selected subsets (ablation) | Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2 · AlpacaEval 2.0 | win rate: more than 10 points | Hurt |
| Reduce the learning rate from 5×10⁻⁷ to 3×10⁻⁷ or early-stop at epoch 1 (ablation) | DPO on Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2 · UltraFeedback | win rate | No clear effect |
| Select the BeeS subset rather than random 6,000 samples (ablation) | IPO, KTO, and SLiC on Llama-3-8B (Base) · AlpacaEval 2.0, Llama-UltraFeedback | win rate: over 12% | Helped |
Surprising findings
- Negative-margin (most negative) samples are learned as quickly as the most positive ones, so 'bad' preferences are easy to fit.
- Using a small beta (0.01) with DPO can cause model breakdown, with win rates dropping by more than 10 points on Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2.
Limitations
- Experiments only up to 14B parameters; 70B+ scale not tested.
- No error bars or repeated runs due to cost.
- Small beta can cause unstable training and model degradation.
Experimental setup
Datasets: UltraFeedback Llama-UltraFeedback Mistral-UltraFeedback Reddit TL;DR Anthropic HH RLHFlow/SFT-OpenHermes-2.5-Standard
Benchmarks: AlpacaEval 2.0 AlpacaEval TL;DR win rate HH win rate
Metrics: win rate length-controlled win rate
Baselines: Random selection Low-Gap External margin (M-Ex) Implicit margin (M-Im) IFD margin Full dataset DPO SimPO beta-DPO alpha-DPO IPO KTO SLiC
Builds on: DPO Online RLHF / iterative DPO IFD (instruction following difficulty) Bradley-Terry model
Closest papers by meaning
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.94)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.93)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.93)
- Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization (NeurIPS 2025 · similarity 0.93)
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.93)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.