Research map / Large language models / Alignment and preferences
RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RainbowPO, a unified DPO-family objective that combines length normalization, reference-policy mixing, and contextual scaling, tuned on Llama3-8B-Instruct to reach 51.66% AlpacaEval2 LC win rate.
Problem
DPO extensions (IPO, KTO, ORPO, SimPO, etc.) are hard to compare under fair setups, and it is unclear which of their added components actually improve over DPO.
Approach
Decomposes DPO variants into seven orthogonal components (length normalization, link function, home advantage/margin, reference policy, contextual scaling, rejection sampling, SFT loss), empirically tests each, and combines the helpful ones into one objective; it also introduces a mixing reference policy between the SFT model and a margin-defined reference.
Key results
- RainbowPO improves Llama3-8B-Instruct AlpacaEval2 LC win rate from 41.88% (base) to 51.66% with 3 epochs (GPT4-turbo judge).
- Under one epoch, RainbowPO reaches 48.08 LC WR vs SimPO 47.96 and DPO 37.95 on GPT4 judge.
- Ablation: removing length normalization drops LC WR from 51.66 to 45.68.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add length normalization | DPO · AlpacaEval2 (GPT4) | LC WR (%): 41.88 -> 44.27 | Helped |
| add reference policy mixing | DPO with length normalization · AlpacaEval2 (GPT4) | LC WR (%): 44.27 -> 47.45 | Helped |
| add contextual scaling | DPO with length normalization · AlpacaEval2 (GPT4) | LC WR (%): 44.27 -> 45.92 | Helped |
| add home advantage | DPO · AlpacaEval2 (GPT4) | LC WR (%): 41.88 -> 41.70 | Hurt |
| replace the logistic link with the square link function | DPO · AlpacaEval2 (GPT4) | LC WR (%): 41.88 -> 39.53 | Hurt |
| add rejection sampling | DPO · AlpacaEval2 (GPT4) | LC WR (%): 41.88 -> 42.87 | Helped |
| add rejection sampling to length-normalized DPO | DPO with length normalization · AlpacaEval2 (GPT4) | LC WR (%): 44.27 -> 43.22 | Hurt |
| add SFT loss to length-normalized DPO | DPO with length normalization · AlpacaEval2 (GPT4) | LC WR (%): 44.27 -> 39.90 | Hurt |
| add reference policy mixing | DPO · AlpacaEval2 (GPT4) | LC WR (%): 41.88 -> 40.18 | Hurt |
| remove reference policy mixing (ablation) | RainbowPO · AlpacaEval2 (GPT4), 3 epochs | LC WR (%): 50.52 -> 51.66 | Helped |
| remove contextual scaling (ablation) | RainbowPO · AlpacaEval2 (GPT4), 3 epochs | LC WR (%): 48.52 -> 51.66 | Helped |
| remove length normalization (ablation) | RainbowPO · AlpacaEval2 (GPT4), 3 epochs | LC WR (%): 45.68 -> 51.66 | Helped |
Surprising findings
- Home advantage (margin) added to DPO consistently degraded performance, contrary to SimPO's claim that the margin is beneficial.
- RSO improves DPO but degrades when combined with length normalization, so mathematically orthogonal components are not empirically independent.
Limitations
- Evaluated only on Llama3-8B-Instruct and AlpacaEval2 instruction following.
- Hyperparameters tuned greedily; many combinations not explored.
- Some component interactions remain unexplained.
Experimental setup
Datasets: UltraFeedback AlpacaEval2
Benchmarks: AlpacaEval2
Metrics: length-controlled win rate win rate average length
Baselines: DPO IPO KTO CPO ORPO SimPO MallowsPO
Builds on: DPO SimPO MallowsPO RSO ORPO
Closest papers by meaning
- MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge (NeurIPS 2026 · similarity 0.90)
- SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024 · similarity 0.89)
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.89)
- Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization (ICLR 2025 · similarity 0.89)
- MallowsPO: Fine-Tune Your LLM with Preference Dispersions (ICLR 2025 · similarity 0.88)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.