Research map / Reinforcement learning / RL from feedback
Stepwise Alignment for Constrained Language Model Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SACPO, a stepwise alignment method that first aligns an LLM to reward and then realigns it to safety with DPO/KTO (plus P-SACPO via model merging), with theoretical optimality and safety-violation bounds, and shows gains over Safe RLHF on Alpaca-7B.
Problem
Aligning LLMs to maximize helpfulness (reward) under a safety constraint, where Safe RLHF requires separate reward/safety models and unstable PPO-Lagrangian training.
Approach
The optimal constrained policy equals a reward-aligned policy realigned to the safety function with parameter beta/lambda*, so reward and safety can be aligned sequentially using RL-free losses (DPO, KTO) on separate datasets, with a conservative fixed multiplier and weight merging to trade off.
Key results
- SACPO variants (e.g., DPO(H)->DPO(S), KTO(H)->DPO(S)) outperform Safe RLHF beaver-7b-v1.0 in both helpfulness and harmlessness win rates against the SFT model on Alpaca-7B.
- Varying beta/lambda traces a near Pareto front between helpfulness and harmlessness.
- P-SACPO with linear model merging performs better than naive weight averaging of DPO(H) and DPO(S).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a first-step DPO helpfulness alignment | SACPO on Alpaca-7B · AlpacaEval helpful_base prompts | helpfulness | Helped |
| Add a first-step KTO helpfulness alignment | SACPO on Alpaca-7B · AlpacaEval helpful_base prompts | helpfulness | Helped |
| Add DPO safety realignment after DPO helpfulness alignment | SACPO on Alpaca-7B · Safe RLHF red-teaming prompts | harmlessness | Helped |
| Add DPO safety realignment after KTO helpfulness alignment | SACPO on Alpaca-7B · Safe RLHF red-teaming prompts | harmlessness | Helped |
| Replace DPO safety realignment with KTO after DPO helpfulness alignment (ablation) | SACPO on Alpaca-7B · Helpfulness and harmlessness evaluations | win rate vs SFT | Hurt |
| Reverse the alignment order to safety then helpfulness (ablation) | SACPO on Alpaca-7B · Helpfulness and harmlessness evaluations | win rate vs SFT | No clear effect |
| Use safety-then-helpfulness order with DPO at β/λ = 0.01 (ablation) | SACPO on Alpaca-7B · Helpfulness and harmlessness evaluations | win rate vs SFT | Mixed |
| Vary the β/λ ratio (ablation) | SACPO on Alpaca-7B · Helpfulness and harmlessness evaluations | helpfulness-harmlessness trade-off | Helped |
| Merge the helpfulness-aligned and conservatively safety-realigned policies | P-SACPO on Alpaca-7B · Helpfulness and harmlessness evaluations | win rate vs SFT | Helped |
Surprising findings
- Alignment order (safety-first vs. helpfulness-first) gives comparable performance in most settings, though gaps appear for DPO(S)->DPO(H) at beta/lambda=0.01.
- DPO(H)->KTO(S) performs significantly worse than DPO(H)->DPO(S), suggesting KTO is less suitable for safety alignment.
Limitations
- Evaluated only on 7B models; unclear scaling to much larger state-of-the-art models.
- Requires substantial computation and large amounts of high-quality data; does not incorporate standard SFT or pre/post-check strategies.
- Theoretical bounds rely on linear reward/safety assumptions and Slater condition; evaluation via GPT-4 judge only.
Experimental setup
Datasets: PKU-SafeRLHF AlpacaEval helpful_base prompts Safe RLHF red-teaming prompts
Benchmarks: AlpacaEval helpful_base (129 prompts) Safe RLHF red-teaming prompts (83 prompts)
Metrics: win rate against SFT model Elo score
Baselines: Safe RLHF (beaver-7b-v1.0) Safe RLHF (beaver-7b-v2.0) Safe RLHF (beaver-7b-v3.0) SFT model naive linear merging
Builds on: DPO KTO Safe RLHF PPO-Lagrangian linear model merging
Closest papers by meaning
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.90)
- One-Shot Safety Alignment for Large Language Models via Optimal Dualization (NeurIPS 2024 · similarity 0.89)
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.89)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.89)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.89)
- AlphaPO: Reward Shape Matters for LLM Alignment (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.