Research map / Large language models / Alignment and preferences
Safe RLHF: Safe Reinforcement Learning from Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Safe RLHF, which decouples helpfulness and harmlessness annotations, trains separate reward and cost models, and uses a Lagrangian-constrained PPO to fine-tune Alpaca-7B over three iterations.
Problem
Helpfulness and harmlessness objectives conflict in LLM alignment; crowdworker preference labels that merge them are confused and single-objective or statically weighted reward shaping over-optimizes one objective.
Approach
Decouple human annotation into independent helpfulness rankings and harmlessness rankings plus binary safety meta-labels, train separate reward and cost models, and cast alignment as a constrained RL problem solved with a dynamically updated Lagrange multiplier instead of a fixed trade-off weight.
Key results
- Beaver-v3 vs Alpaca-7B: helpfulness Elo +244.91 (GPT-4) and +363.86 (human); harmlessness Elo +268.31 (GPT-4) and +237.98 (human).
- Probability of harmful responses on evaluation set fell from 53.08% (Alpaca-7B) to 2.45% (Beaver-v3) per human labels.
- Reward shaping with fixed weights either over-optimized one objective or underperformed Safe RLHF; decoupled annotation raised inter-rater agreement to 69.00% helpfulness and 66.53% safety vs 61.65% for merged labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply three rounds of Safe RLHF | Alpaca-7B · Helpfulness Elo, GPT-4 evaluation | Elo score change: +244.91 | Helped |
| Apply three rounds of Safe RLHF | Alpaca-7B · Harmlessness Elo, GPT-4 evaluation | Elo score change: +268.31 | Helped |
| Apply three rounds of Safe RLHF | Alpaca-7B · Helpfulness Elo, human evaluation | Elo score change: +363.86 | Helped |
| Apply three rounds of Safe RLHF | Alpaca-7B · Harmlessness Elo, human evaluation | Elo score change: +237.98 | Helped |
| Apply three rounds of Safe RLHF | Alpaca-7B · Evaluation set | Probability of harmful responses: 53.08% -> 2.45% | Helped |
| Decouple helpfulness and harmlessness annotation and training (ablation) | Conventional single-dimensional RLHF · Preference annotation | Inter-Rater Agreement Rate: Helpfulness: 69.00% and Safety: 66.53% compared to 61.65% | Helped |
| Use conventional single-dimensional annotation and PPO training instead of decoupling (ablation) | Safe RLHF · Helpfulness | Helpfulness | Helped |
| Use conventional single-dimensional annotation and PPO training instead of decoupling (ablation) | Safe RLHF · Harmlessness | Harmlessness | Hurt |
| Replace dynamic Lagrangian balancing with fixed reward shaping (ablation) | Safe RLHF · Helpfulness and harmlessness | Win rates against Alpaca-7B | Hurt |
| Use safety-classifier logits as cost signals instead of the Cost Model (ablation) | Safe RLHF · Harmlessness | Harmlessness | Hurt |
| Remove Cost Model classification capability and stop updating Lagrange multipliers (ablation) | Safe RLHF · Helpfulness and harmlessness | Objective balance | Hurt |
Surprising findings
- Reward shaping with moderate fixed weights (nu=1,2) still underperformed Safe RLHF, and extreme weights over-optimized one objective at the other's expense.
- A cost model trained on classifier logits as cost signal was significantly less effective at improving harmlessness than the proposed cost model.
Limitations
- Relies on the inaccessible pretraining data, substituting Alpaca SFT data for PTX loss; limited high-quality SFT data.
- Only single-turn conversations and two dimensions (helpfulness and harmfulness) are considered.
- Substantial financial cost of crowdsourced annotation and RLHF training.
Experimental setup
Datasets: Stanford Alpaca dataset (PTX loss) open-source safety-related red-teaming prompt datasets crowdworker preference data (helpfulness and harmlessness)
Benchmarks: Safe RLHF evaluation prompt set (14 safety categories, open-source prompts, red-team prompts)
Metrics: Elo score (helpfulness and harmlessness) win rate vs SFT reward and cost model scores probability of harmful response (human-labeled) ranking accuracy and safety classification accuracy of rewar inter-rater agreement
Baselines: Alpaca-7B Reward Shaping conventional RLHF (PPO with single preference) classifier-logit cost signal (CM-classifier)
Builds on: RLHF PPO Bradley-Terry preference model Constrained MDP / Safe RL Alpaca
Closest papers by meaning
- Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback (NeurIPS 2025 · similarity 0.93)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.92)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.92)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.92)
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.92)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.