AI Paper Atlas

Research map / Large language models / Alignment and preferences

Data Selection for LLM Alignment Using Fine-Grained Preferences

Jia Zhang, Yao Liu, Chen-Xi Zhang, Yi Liu, Yi-Xuan Jin, Lan-Zhe Guo, Yu-Feng Li

ICLR 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
select samples with most negative PD values instead of the largest PD values (ablation)PD selection · Taobao Live (30% budget)win score vs SFT: 0.48 -> 1.43Improved
select samples with most negative PD values instead of middle PD values (ablation)PD selection · Taobao Live (30% budget)win score vs SFT: 1.33 -> 1.43Improved
add explicit length bias mitigation to proxy reward models (ablation)PD selection · UltraFeedback / AlpacaEval 2win rateImproved
select 30% via PD instead of random (ablation)DPO on Llama3.1-8B · UltraFeedback / AlpacaEval 2length-controlled win rate (%): 19.56 -> 26.11Improved
select 30% via PD instead of training on all data (ALL)DPO on Llama3.1-8B · UltraFeedback / AlpacaEval 2length-controlled win rate (%): 21.14 -> 26.11Improved
increase conflict level from 10% to 30% when training on all dataALL (DPO) · UltraFeedback / AlpacaEval 2length-controlled win rate (%): 21.14 -> 16.44Hurt
increase conflict level from 10% to 30% with PD selectionPD (ours) · UltraFeedback / AlpacaEval 2length-controlled win rate (%): 26.11 -> 24.71No clear effect
use PD (ours) instead of ground-truth ratings PD (rati.)PD selection · HelpSteer / AlpacaEval 2length-controlled win rate (%): 12.41 vs 12.13No clear effect
use DMPO over full data instead of OVA.DPO on Llama3.1-8B · UltraFeedback / AlpacaEval 2length-controlled win rate (%): 19.96 -> 24.73Improved
use PD selection instead of DMPO on full data, cutting GPU hoursDPO on Llama3.1-8B · UltraFeedback / AlpacaEval 2GPU hours: 42.1 -> 18.6Improved
set length penalty rho=0 or excessively large (ablation)PD selection · UltraFeedbackwin rateHurt
set gamma=1 (no quantile normalization) (ablation)PD selection · UltraFeedbackwin rateHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy