AI Paper Atlas

Research map / Large language models / Alignment and preferences

When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets

Aladin Djuhera, Farhan Ahmed, Swanand Ravindra Kadhe, Syed Zawad, Heiko Ludwig, Holger Boche

ICLR 2026 · Large language models › Alignment and preferences · approach: DPO design choices · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace TuluDPO with UltraMix-190k for DPOLlama-3.1-8B-TuluSFT · Overall averagescore: 53.96 -> 56.04Improved
replace TuluDPO with UltraMix-190k for DPOQwen-2.5-7B-TuluSFT · Overall averagescore: 59.56 -> 62.05Improved
initial quality/reward filtering recipe UltraMix-170k instead of TuluDPOLlama-3.1-8B-TuluSFT · Overall averagescore: 53.96 -> 54.16Improved
UltraMix-170k instead of TuluDPO on TruthfulQALlama-3.1-8B-TuluSFT · TruthfulQAscore: 56.78 -> 61.45Improved
UltraMix-170k instead of TuluDPO on MATHQwen-2.5-7B-TuluSFT · MATHscore: 43.13 -> 47.55Improved
UltraMix-170k instead of TuluDPO on code and IFEvalboth models · HumanEval / IF-EvalscoreHurt
add task-aware retention of instruction-following samples to the UltraMix recipe (170k to 190k)UltraMix · Overall averagescore: 54.16 -> 56.04Improved
use UltraMix instead of RIP-style filtering (ablation)Llama-3.1-8B-TuluSFT · MMLUscore: 62.93 -> 64.61Improved
use UltraMix instead of RIP-style filtering (ablation)Qwen-2.5-7B-TuluSFT · GSM8Kscore: 76.85 -> 82.70Improved
train DPO on preference-reward-filtered HelpSteer3 instead of base (ablation)Llama-3.1-8B-TuluSFT · HelpSteer3 averagescore: 51.53 -> 52.25Improved
train DPO on RIP-filtered HelpSteer3 instead of base (ablation)Llama-3.1-8B-TuluSFT · HelpSteer3 averagescore: 51.53 -> 50.45Hurt
train DPO on preference-reward-filtered HelpSteer3 instead of base (ablation)Qwen-2.5-7B-TuluSFT · HelpSteer3 averagescore: 58.46 -> 58.83Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy