Research map / Large language models / Alignment and preferences
Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes that overly difficult preference examples (high DPO validation loss) hinder alignment, and introduces Selective DPO, which trains only on the easiest ~50% of examples ordered by validation loss.
Problem
Preference alignment (DPO) typically assumes more clean data is better, ignoring the match between example difficulty and model capacity; overly difficult preference examples can degrade alignment.
Approach
Defines example difficulty via validation loss from reference DPO models trained on held-out data partitions, shows difficult examples hurt DPO and that the optimal data percentage grows with model size, and proposes Selective DPO, which filters out the hardest examples and trains from easy to difficult.
Key results
- Selective DPO improves AlpacaEval 2 win rate over DPO by roughly 9-16% (e.g., Mistral-7B WR 27.1 vs DPO 12.5 with LC 25.4 vs 15.1 when using full FPFT configuration).
- On Llama-3-8B-SFT, Selective DPO reaches 24.9 LC and 25.3 WR on AlpacaEval 2 versus DPO 18.2 LC and 15.5 WR.
- The sweet spot of manageable data percentage rises from 64% for Qwen-2.5-3B to 81% for Qwen-2.5-14B on Argilla-dpo-mix-7k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Filter out overly difficult examples using validation loss | DPO alignment · AlpacaEval 2 | win rate: 9-16% higher win rate | Helped |
| Include overly difficult examples in training | DPO alignment · AlpacaEval 2 | win rate: 9.4% | Hurt |
| Use only easy examples, sorted by validation loss, instead of random-order DPO | DPO alignment · UltraFeedback-binarized and Argilla-dpo-mix-7k | WR$'$ | Helped |
| Shuffle the selected easy examples rather than sorting easy-to-difficult (ablation) | Selective DPO · UltraFeedback-binarized and Argilla-dpo-mix-7k | WR$'$ | No clear effect |
| Flip labels of the most difficult 40% of examples (ablation) | DPO alignment · UltraFeedback-binarized | WR$'$ | No clear effect |
| Mix randomly sampled examples into validation-loss ordering with epsilon-greedy sorting (ablation) | DPO alignment · Difficult-example alignment experiments | WR$'$ | No clear effect |
| Adjust the learning rate (ablation) | DPO alignment with difficult examples · Difficult-example alignment experiments | WR$'$ | No clear effect |
| Raise the capacity from Qwen-2.5-3B to Qwen-2.5-14B | DPO alignment · Argilla-dpo-mix-7k | optimal selected-data percentage: 64% to 81% | Helped |
| Increase the number of reference models used to estimate validation loss (ablation) | Selective DPO · AlpacaEval 2 | length-controlled win rate | Helped |
| Increase the selected-data fraction beyond the model's capacity (ablation) | Selective DPO · AlpacaEval 2 | performance | Hurt |
| Select only the simplest examples by setting the selected-data fraction too low (ablation) | Selective DPO · AlpacaEval 2 | performance | Hurt |
| Use a curriculum derived from a 3B reference model instead of the target 7B model (ablation) | 7B DPO alignment · AlpacaEval 2 | performance | Hurt |
Surprising findings
- Label flipping of the hardest 40% examples does not recover performance, suggesting difficult examples are not primarily mislabeled data.
- Selective DPO drops GSM8K accuracy notably on Mistral-7B (30.4 for DPO vs 14.48 for Selective DPO), attributed to excluded difficult math examples and formatting issues.
Limitations
- Selective DPO tends to favor longer responses due to potential data bias.
- The principle is validated only for the DPO setting and not directly for RLHF.
- Performance on some downstream tasks (e.g., GSM8K for Mistral-7B) drops.
Experimental setup
Datasets: UltraChat-200k UltraFeedback-binarized Argilla-dpo-mix-7k
Benchmarks: AlpacaEval 2 Arena-Hard v0.1 MT-Bench HuggingFace Open Leaderboard
Metrics: win rate length-controlled win rate MT-Bench score accuracy
Baselines: DPO label flipping label smoothing IPO KTO ORPO SimPO WPO R-DPO CHES RM filtering PPL filtering
Builds on: DPO curriculum learning learned step difficulty measure
Closest papers by meaning
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.91)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.91)
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.91)
- Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.91)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.91)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.