Research map / Large language models / Alignment and preferences
Keep the Best, Forget the Rest: Reliable Alignment with Order-Aware Preference Optimization
Jiahui Zhu, Yuanjie Shi, Xiyue Peng, Xin Liu, Yan Yan, Honghao Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| filter out the 1 highest-loss sample per batch (RAPPO-1) vs DPO | DPO · IMDb sentiment (GPT2-Large) | reward score: 1.5513 -> 1.6625 | Improved |
| filter out 4 highest-loss samples per batch (RAPPO-4) vs DPO | DPO · IMDb sentiment (GPT2-Large) | reward score: 1.5513 -> 1.6811 | Improved |
| add order-aware filtering (RAPPO-4) vs DPO | DPO · Toxicity (GPT-Neo-2.7B) | toxicity (%): 6.30 -> 2.28 | Improved |
| RAPPO-4 vs SimPO | SimPO · IMDb sentiment | reward score: 1.5537 -> 1.6811 | Improved |
| raise filtered q from 1 to 2 (ablation) | RAPPO · IMDb batch 32 tau=1.0 | reward score: 1.6625 -> 1.6790 | Improved |
| raise filtered q from 4 to 8 (over-filtering) (ablation) | RAPPO · IMDb batch 32 tau=1.0 | reward score: 1.6811 -> 1.6432 | Hurt |
| vary q across 1,2,4,8 against SimPO (ablation) | RAPPO · IMDb batch 16 and 32 | reward score: 1.6600 -> 1.7481 | No clear effect |
| vary threshold tau from 0.8 to 1.2 at q=4 (ablation) | RAPPO · IMDb batch 32 | reward score: 1.6765 -> 1.6828 | No clear effect |
| use RAPPO instead of DPO | DPO · PKU-SafeRLHF (Mistral-7B) | safety rate: 55.89% -> 57.26% | Improved |
| use RAPPO instead of DPO | DPO · PKU-SafeRLHF (Mistral-7B) | Beaver-Help helpfulness: 0.51 -> 0.69 | Improved |
| use RAPPO instead of DPO | DPO · PKU-SafeRLHF (Mistral-7B) | Beaver-Harm: 0.45 -> 0.36 | Improved |
| RAPPO vs SimPO under four LLM judges | SimPO · Summarization (Llama-3.1-8B) | win rate: 58.30% to 72.00% | Improved |
Closest papers by meaning
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.94)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.93)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.93)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.93)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.93)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
