Research map / Large language models / Alignment and preferences
Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rectified Policy Optimization (RePO), which replaces expected safety constraints with per-prompt critical safety constraints via a rectified penalty, and shows gains in safety alignment of Alpaca-7B and Llama3.2-3B over SafeRLHF and SACPO.
Problem
Expected safety constraints in safe RLHF for LLMs can be satisfied on average while individual prompt-response pairs remain unsafe ("safety compensation"), so the safety alignment is incomplete.
Approach
Replace the expected cost constraint with a critical constraint requiring C(x,y)<=0 for all prompt-response pairs, reformulated as a min-max problem with rectified cost {C}^+ and a non-decreasing penalty, optimized via a PPO-style actor-critic that splits batches into safe and unsafe subsets.
Key results
- On Alpaca-7B, RePO reaches a 96.08% safety rate in model-based evaluation versus 90.58% for SafeRLHF and 80.72% for SACPO, with +1.01 helpfulness delta.
- On Llama3.2-3B, RePO achieves a 91.46% safety rate and 68.60% GPT-4 safety rate, versus 76.74% and 45.91% for SafeRLHF.
- In GPT-4o pairwise evaluation on Alpaca-7B, RePO wins 67.64% against the initial model versus 7.95% losses.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Alpaca-7B · PKU-SafeRLHF test | Safety rate: 90.58% -> 96.08% | Helped |
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Llama3.2-3B · PKU-SafeRLHF test | Safety rate: 76.74% -> 91.46% | Helped |
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Alpaca-7B · PKU-SafeRLHF test | Harmlessness (average cost): -12.50 -> -13.85 | Helped |
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Llama3.2-3B · PKU-SafeRLHF test | Harmlessness (average cost): -6.92 -> -12.43 | Helped |
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Alpaca-7B · PKU-SafeRLHF test | Δ helpfulness: -0.71 -> +1.01 | Helped |
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Llama3.2-3B · PKU-SafeRLHF test | Δ helpfulness: -1.20 -> +0.16 | Mixed |
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Alpaca-7B · GPT-4 evaluation | Safety rate: 51.16% -> 72.03% | Helped |
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Llama3.2-3B · GPT-4 evaluation | Safety rate: 45.91% -> 68.60% | Helped |
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Alpaca-7B · PKU-SafeRLHF test | Safety rate: 80.72% -> 96.08% | Helped |
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Llama3.2-3B · PKU-SafeRLHF test | Safety rate: 53.73% -> 91.46% | Helped |
| replace expected safety constraints with critical safety constraints using RePO (ablation) | Llama3.2-3B · PKU-SafeRLHF test | Δ helpfulness: +2.90 -> +0.16 | Hurt |
Surprising findings
- SACPO achieves the highest helpfulness gain on Llama3.2-3B (+2.90) yet has a positive average cost (4.12) and only 53.73% safety rate.
- Expected-constraint baselines leave roughly one-third of samples per batch unsafe even after fine-tuning, according to the appendix analysis.
Limitations
- Evaluation relies largely on beaver reward/cost models and GPT-4o as judge, which may be biased.
- Experiments cover only two base models (Alpaca-7B and Llama3.2-3B) and one safety dataset family.
- No formal convergence guarantee is given for the actor-critic approximation of the rectified gradient.
Experimental setup
Datasets: PKU-SafeRLHF Alpaca PhysicalSafety CoNa Controversial MaliciousInstructions
Benchmarks: PKU-SafeRLHF test set PhysicalSafety CoNa Controversial MaliciousInstructions
Metrics: helpfulness reward delta harmlessness cost safety rate GPT-4 win rate GPT-4 safety rate
Baselines: SafeRLHF SACPO
Builds on: PPO-Lagrangian Safe RLHF SACPO PPO DPO
Closest papers by meaning
- BSO: Safety Alignment Is Density Ratio Matching (NeurIPS 2026 · similarity 0.93)
- One-Shot Safety Alignment for Large Language Models via Optimal Dualization (NeurIPS 2024 · similarity 0.92)
- AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.92)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.92)
- Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback (NeurIPS 2025 · similarity 0.90)
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.