Research map / Reinforcement learning / RL from feedback
Implicit Safety Alignment from Crowd Preferences
Qian Lin, Daniel S. Brown
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compose preference-aligned skills with a high-level policy instead of combining rewards (Safe-VPL) | downstream policy learning · 6 safe RL envs (avg) | normalized cost: 1.00 -> 0.03 | Improved |
| Safe-CPL skill composition vs Task-Only | downstream policy learning · 6 safe RL envs (avg) | normalized reward: 1.46 -> 0.92 | Hurt |
| reward combination RC(w=0.5) on coupled crowd preferences instead of safety-only preferences (SOPL) (ablation) | reward combination · 6 safe RL envs (avg) | normalized reward: 1.04 -> 0.82 | Hurt |
| imbalanced crowd preferences (one user contributes 10x data) (ablation) | reward combination · safe RL envs | reward and cost frontier: degrades by at least 0.1 (RC averaged over w) | Hurt |
| imbalanced crowd preferences (one user contributes 10x data) (ablation) | Safe-VPL/Safe-CPL · safe RL envs | normalized reward and cost: decrease less than 0.02 | No clear effect |
| vary trade-off weight w in reward combination (ablation) | reward combination · safe RL envs | reward vs cost | Hurt |
| increase prior regularization weight beta_reg (ablation) | Safe crowd preference RL · safe RL envs | reward and cost | Mixed |
| increase preference label noise (ablation) | Safe crowd preference RL · safe RL envs | safety cost and task reward | Mixed |
| increase number of crowd users (ablation) | Safe crowd preference RL · safe RL envs | task performance and cost | Mixed |
| apply skill composition to LLM-style bandit instead of Task-only | Safe crowd preference RL · LLM-style conversational bandit | cost: 0.24 -> 0.00 | Improved |
| RC with w=0.75 in LLM-style bandit under imbalanced preferences (ablation) | reward combination · LLM-style conversational bandit | reward: 0.95 -> 0.50 | Hurt |
| use safety-only preference data (SOPL) rather than entangled crowd preferences (ablation) | reward combination · 6 safe RL envs (avg) | normalized cost: 0.05 -> 0.01 | Improved |
Closest papers by meaning
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.89)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.89)
- AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.88)
- BSO: Safety Alignment Is Density Ratio Matching (NeurIPS 2026 · similarity 0.88)
- RuleAdapter: Dynamic Rules for training Safety Reward Models in RLHF (ICML 2025 · similarity 0.88)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
