Research map / Large language models / Alignment and preferences
Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sequential Preference Optimization (SPO), which aligns an LLM to multiple preference dimensions in sequential DPO-style rounds with a KL-style constraint preserving earlier dimensions, avoiding explicit reward models.
Problem
Aligning LLMs with multi-dimensional human preferences (e.g., helpfulness and harmlessness) that may conflict, without the complexity of training and balancing multiple reward models as in RLHF.
Approach
Sequentially fine-tune one preference dimension per round with DPO-like implicit rewards; the closed-form optimal policy yields a reward for round n as a weighted sum of log-ratios from previous rounds, so the loss constrains preservation of earlier-dimension alignment without any explicit reward model.
Key results
- On Llama 2 7B with PKU-SafeRLHF modified data, SPO beats S-DPO on HH-helpful (81.5% for Safe-RLHF baseline vs 53.1% etc. reported as SPO win rates); SPO win rate against DPO-Mix on AlpacaEval is 77.9%.
- After four-dimension sequential tuning on the synthetic token task, SPO reaches 50.3% Pareto-optimal responses while S-DPO reaches 0%.
- Against merged DPO models on AlpacaEval, SPO wins 79.0%, but loses on PKU-SafeRLHF-Test (41.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPO to sequentially optimize helpfulness and harmlessness instead of unconstrained S-DPO (ablation) | Llama 2 7B · HH-helpful and HH-harmless | win rate against S-DPO: 81.5% (HH-helpful); 41.9% (HH-harmless) | Mixed |
| Use SPO to sequentially optimize helpfulness and harmlessness instead of unconstrained S-DPO (ablation) | Llama 2 13B · HH-helpful and HH-harmless | win rate against S-DPO: 81.5% (HH-helpful); 57.5% (HH-harmless) | Mixed |
| Set α to 0, removing the constraint that preserves prior alignment (ablation) | SPO, Llama 2 7B · Second-round fine-tuning | helpfulness reward: 6.82 (α=0.1) vs 1.42 (α=0) | Hurt |
| Increase α to preserve more of the previous helpfulness alignment (ablation) | SPO, Llama 2 7B · Second-round fine-tuning | helpfulness reward: 1.42; 6.53; 6.82; 6.93; 7.05; 7.13 | Helped |
| Increase α to preserve more of the previous helpfulness alignment (ablation) | SPO, Llama 2 7B · Second-round fine-tuning | harmlessness reward: 2.29; 1.30; 1.18; 0.22; -0.74; -1.42 | Hurt |
| Remove constraints preserving earlier preference dimensions (ablation) | SPO (S-DPO ablation) · Four preference dimensions, demonstrative dataset | Pareto-optimal responses: 50.3% vs 0 | Hurt |
| Remove constraints preserving earlier preference dimensions (ablation) | SPO (S-DPO ablation) · Safe questions during second-round harmlessness fine-tuning | overfitting rate: 0% vs 62.5% | Hurt |
| Apply sequential optimization across four dimensions rather than unconstrained S-DPO | Llama 2 on Helpsteer2 · Helpsteer2 validation, helpfulness, correctness, coherence, and verbosity | win rate against S-DPO: 57.3%; 55.3%; 57.6%; 49.4% | Mixed |
| Use SPO instead of Safe-RLHF and RLHF-RS | Llama 2 7B and 13B · Helpfulness and harmlessness evaluations | win rate against baselines: 77.7%; 64.9%; 58.1%; 55.8%; 80.8%; 67.6%; 58.0%; 56.5%; 82.8%; 74.4%; 53.1%; 51.3%; 85.3%; 73.4%; 75.0%; 78.9% | Helped |
| Fine-tune only on helpfulness or only on harmlessness instead of sequentially aligning both dimensions | Llama 2 · Helpfulness and harmlessness evaluations | win rate against single-dimension DPO: 47.9%; 47.7%; 63.3%; 65.8%; 64.6%; 68.6%; 48.1%; 46.3%; 47.2%; 44.1%; 76.1%; 77.6%; 64.0%; 66.3%; 51.8%; 51.0% | Mixed |
| Merge four dimension-specific DPO models instead of sequentially optimizing dimensions with SPO | Llama 2 on Helpsteer2 · Helpsteer2 validation, four dimensions | win rate against merged DPO: 56.2%, 56.0%, 70.1%, and 61.0% | Mixed |
Surprising findings
- Naive sequential DPO (S-DPO) overfits to the last dimension, e.g. 13B model produces extremely simple harmless responses, degrading helpfulness.
- Merging helpful and harmless RMs performs poorly on both dimensions (51.2% and 48.2% validation accuracy).
Limitations
- Only 7B and 13B Llama 2 models tested; larger state-of-the-art models not evaluated due to compute.
- Evaluation relies on GPT-4 as judge with win rates on a limited set of datasets.
Experimental setup
Datasets: PKU-SafeRLHF-30k Helpsteer2 Alpaca HH-RLHF AlpacaEval PKU-SafeRLHF-test
Benchmarks: HH-helpful AlpacaEval HH-harmless PKU-SafeRLHF-test Helpsteer2 validation
Metrics: win rate Pareto-optimal rate aggregated utility reward model accuracy
Baselines: Safe-RLHF RLHF-RS DPO-Mix Sequential DPO DPO-HP DPO-HM Alpaca Vicuna-7B-v1.5 Mistral-7B-Instruct-v0.1 MODPO
Builds on: DPO RLHF Safe RLHF MODPO
Closest papers by meaning
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.93)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.93)
- Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.93)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.92)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.92)
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.