AI Paper Atlas

Research map / Large language models / Alignment and preferences

BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset

Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, Yaodong Yang

NeurIPS 2023 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 35 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train separate reward and cost models from decoupled helpfulness/harmlessness rankings with PPO-Lagrangian instead of a single mixed-preference PPO (ablation)Alpaca-7B · win rate vs Alpaca-7B, harmlessnesswin rate: 68.64% -> 82.57%Improved
decoupled preference vs mixed preference (ablation)Alpaca-7B · win rate vs Alpaca-7B, helpfulnesswin rate: 65.07% -> 85.57%Improved
use cost model trained on rankings and sign instead of ensemble of 14 binary classifiers (max) (ablation)Safe-RLHF · win rate vs Alpaca-7B, harmlessnesswin rate: 64.50% -> 82.57%Improved
use cost model rather than classifier ensemble (max) (ablation)Safe-RLHF · win rate vs Alpaca-7B, helpfulnesswin rate: 74.00% -> 85.57%Improved
use max instead of mean of classifier probabilities as cost (ablation)PPOL-classifier · win rate vs Alpaca-7B, harmlessnesswin rate: 59.07% -> 64.50%Improved
train on BeaverTails decoupled data instead of HH-RLHF dataset (ablation)PPO-based RLHF · win rate vs Alpaca-7B, harmlessnesswin rate: 66.21% -> 82.57%Improved
train on BeaverTails decoupled data instead of HH-RLHF dataset (ablation)PPO-based RLHF · win rate vs Alpaca-7B, helpfulnesswin rate: 64.93% -> 85.57%Improved
two-stage annotation (14 harm categories then ranking) instead of single-stage (ablation)annotation process · annotator-researcher agreementagreement rate: approximately 15% increaseImproved
apply Safe RLHF fine-tuningAlpaca-7B · static cost model score on red-team promptscostImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy