Research map / Large language models / Alignment and preferences
Towards Understanding Valuable Preference Data for Large Language Model Alignment
Zizhuo Zhang, Qizhou Wang, Shanshan Ye, Jianing Zhu, Jiangchao Yao, Bo Han, Masashi Sugiyama
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LossDiff-IRM selection (about 50-64% of data) instead of full data | DPO · Llama-3.1-8B, AlpacaEval | WinRate: 78.41 -> 87.08 | Improved |
| select data with LossDiff-IRM instead of full data | DPO · Qwen3-8B-Base, AlpacaEval | WinRate: 63.85 -> 71.52 | Improved |
| train only on the pairs dropped by LossDiff-IRM instead of full data (ablation) | DPO · Llama-3.1-8B, AlpacaEval | WinRate: 78.41 -> 61.15 | Hurt |
| train only on dropped pairs instead of full data (ablation) | SLiC · Qwen3-8B-Base, Vicuna-Bench | WinRate: 54.69 -> 56.72 | Hurt |
| use LossDiff-IRM selection instead of the best baseline | DPO · Llama-3.1-8B, four benchmarks | WinRate: +6.45% on average | Improved |
| use LossDiff-IRM selection instead of the best baseline | DPO · Llama-3.1-8B and Qwen3-8B-Base | WinRate: +2.62% | Improved |
| combine LossDiff and IRM instead of LossDiff alone or IRM alone (ablation) | LossDiff-IRM selection · DPO and SLiC | WinRate | Improved |
| combine LossDiff and IRM instead of either alone (ablation) | LossDiff-IRM · Qwen3-0.6B-Base epoch 1, overlap with exact TIF | overlap coefficient: 0.6953 -> 0.7820 | Improved |
| replace exact IF with LossDiff-IRM for scoring | data selection · Llama-3.2-1B probing set | compute time: about 10 hours -> about 5 minutes | Improved |
| flip validation labels at increasing noise rate (ablation) | LossDiff-IRM · validation noise 0.1 to 0.4 | performance | Hurt |
| tighten the percentile thresholds to filter more data (ablation) | LossDiff-IRM · Llama-3.1-8B DPO, four benchmarks | performance | Mixed |
| train on small-IF data instead of medium-IF data (ablation) | DPO · Qwen3-0.6B-Base and Llama-3.2-1B probing set | eval loss and reward margin | Hurt |
Closest papers by meaning
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.93)
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.93)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.93)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.93)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.93)
- Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples (ICML 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
