Research map / Large language models / Alignment and preferences
Data Selection for LLM Alignment Using Fine-Grained Preferences
Jia Zhang, Yao Liu, Chen-Xi Zhang, Yi Liu, Yi-Xuan Jin, Lan-Zhe Guo, Yu-Feng Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select samples with most negative PD values instead of the largest PD values (ablation) | PD selection · Taobao Live (30% budget) | win score vs SFT: 0.48 -> 1.43 | Improved |
| select samples with most negative PD values instead of middle PD values (ablation) | PD selection · Taobao Live (30% budget) | win score vs SFT: 1.33 -> 1.43 | Improved |
| add explicit length bias mitigation to proxy reward models (ablation) | PD selection · UltraFeedback / AlpacaEval 2 | win rate | Improved |
| select 30% via PD instead of random (ablation) | DPO on Llama3.1-8B · UltraFeedback / AlpacaEval 2 | length-controlled win rate (%): 19.56 -> 26.11 | Improved |
| select 30% via PD instead of training on all data (ALL) | DPO on Llama3.1-8B · UltraFeedback / AlpacaEval 2 | length-controlled win rate (%): 21.14 -> 26.11 | Improved |
| increase conflict level from 10% to 30% when training on all data | ALL (DPO) · UltraFeedback / AlpacaEval 2 | length-controlled win rate (%): 21.14 -> 16.44 | Hurt |
| increase conflict level from 10% to 30% with PD selection | PD (ours) · UltraFeedback / AlpacaEval 2 | length-controlled win rate (%): 26.11 -> 24.71 | No clear effect |
| use PD (ours) instead of ground-truth ratings PD (rati.) | PD selection · HelpSteer / AlpacaEval 2 | length-controlled win rate (%): 12.41 vs 12.13 | No clear effect |
| use DMPO over full data instead of OVA. | DPO on Llama3.1-8B · UltraFeedback / AlpacaEval 2 | length-controlled win rate (%): 19.96 -> 24.73 | Improved |
| use PD selection instead of DMPO on full data, cutting GPU hours | DPO on Llama3.1-8B · UltraFeedback / AlpacaEval 2 | GPU hours: 42.1 -> 18.6 | Improved |
| set length penalty rho=0 or excessively large (ablation) | PD selection · UltraFeedback | win rate | Hurt |
| set gamma=1 (no quantile normalization) (ablation) | PD selection · UltraFeedback | win rate | Hurt |
Closest papers by meaning
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.94)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.93)
- Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples (ICML 2025 · similarity 0.93)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.93)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.93)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
