Research map / Large language models / Alignment and preferences
When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
Aladin Djuhera, Farhan Ahmed, Swanand Ravindra Kadhe, Syed Zawad, Heiko Ludwig, Holger Boche
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace TuluDPO with UltraMix-190k for DPO | Llama-3.1-8B-TuluSFT · Overall average | score: 53.96 -> 56.04 | Improved |
| replace TuluDPO with UltraMix-190k for DPO | Qwen-2.5-7B-TuluSFT · Overall average | score: 59.56 -> 62.05 | Improved |
| initial quality/reward filtering recipe UltraMix-170k instead of TuluDPO | Llama-3.1-8B-TuluSFT · Overall average | score: 53.96 -> 54.16 | Improved |
| UltraMix-170k instead of TuluDPO on TruthfulQA | Llama-3.1-8B-TuluSFT · TruthfulQA | score: 56.78 -> 61.45 | Improved |
| UltraMix-170k instead of TuluDPO on MATH | Qwen-2.5-7B-TuluSFT · MATH | score: 43.13 -> 47.55 | Improved |
| UltraMix-170k instead of TuluDPO on code and IFEval | both models · HumanEval / IF-Eval | score | Hurt |
| add task-aware retention of instruction-following samples to the UltraMix recipe (170k to 190k) | UltraMix · Overall average | score: 54.16 -> 56.04 | Improved |
| use UltraMix instead of RIP-style filtering (ablation) | Llama-3.1-8B-TuluSFT · MMLU | score: 62.93 -> 64.61 | Improved |
| use UltraMix instead of RIP-style filtering (ablation) | Qwen-2.5-7B-TuluSFT · GSM8K | score: 76.85 -> 82.70 | Improved |
| train DPO on preference-reward-filtered HelpSteer3 instead of base (ablation) | Llama-3.1-8B-TuluSFT · HelpSteer3 average | score: 51.53 -> 52.25 | Improved |
| train DPO on RIP-filtered HelpSteer3 instead of base (ablation) | Llama-3.1-8B-TuluSFT · HelpSteer3 average | score: 51.53 -> 50.45 | Hurt |
| train DPO on preference-reward-filtered HelpSteer3 instead of base (ablation) | Qwen-2.5-7B-TuluSFT · HelpSteer3 average | score: 58.46 -> 58.83 | Improved |
Closest papers by meaning
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.94)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.94)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.93)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.93)
- Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization (ICLR 2025 · similarity 0.92)
- MallowsPO: Fine-Tune Your LLM with Preference Dispersions (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
