AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Implicit Safety Alignment from Crowd Preferences

Qian Lin, Daniel S. Brown

ICML 2026 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
compose preference-aligned skills with a high-level policy instead of combining rewards (Safe-VPL)downstream policy learning · 6 safe RL envs (avg)normalized cost: 1.00 -> 0.03Improved
Safe-CPL skill composition vs Task-Onlydownstream policy learning · 6 safe RL envs (avg)normalized reward: 1.46 -> 0.92Hurt
reward combination RC(w=0.5) on coupled crowd preferences instead of safety-only preferences (SOPL) (ablation)reward combination · 6 safe RL envs (avg)normalized reward: 1.04 -> 0.82Hurt
imbalanced crowd preferences (one user contributes 10x data) (ablation)reward combination · safe RL envsreward and cost frontier: degrades by at least 0.1 (RC averaged over w)Hurt
imbalanced crowd preferences (one user contributes 10x data) (ablation)Safe-VPL/Safe-CPL · safe RL envsnormalized reward and cost: decrease less than 0.02No clear effect
vary trade-off weight w in reward combination (ablation)reward combination · safe RL envsreward vs costHurt
increase prior regularization weight beta_reg (ablation)Safe crowd preference RL · safe RL envsreward and costMixed
increase preference label noise (ablation)Safe crowd preference RL · safe RL envssafety cost and task rewardMixed
increase number of crowd users (ablation)Safe crowd preference RL · safe RL envstask performance and costMixed
apply skill composition to LLM-style bandit instead of Task-onlySafe crowd preference RL · LLM-style conversational banditcost: 0.24 -> 0.00Improved
RC with w=0.75 in LLM-style bandit under imbalanced preferences (ablation)reward combination · LLM-style conversational banditreward: 0.95 -> 0.50Hurt
use safety-only preference data (SOPL) rather than entangled crowd preferences (ablation)reward combination · 6 safe RL envs (avg)normalized cost: 0.05 -> 0.01Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy