Research map / Large language models / Alignment and preferences
More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically evaluates how RLHF (PPO and DPO after SFT) on the Anthropic HH preference data affects five trustworthiness aspects in Pythia and Llama models, and proposes adapting DataInf influence-function attribution to attribute trustworthiness changes to fine-tuning data.
Problem
General-purpose human preference RLHF is assumed to improve LLM trustworthiness, but its effect on toxicity, stereotypical bias, machine ethics, truthfulness, and privacy has not been rigorously evaluated in controlled settings.
Approach
Stage-wise evaluation of SFT, PPO, and DPO on the same base models, plus an adaptation of efficient influence-function data attribution (DataInf) to SFT, PPO reward-model, and DPO losses to trace which preference samples drive trustworthiness changes.
Key results
- RLHF improves machine ethics FNR, reducing it from 56.8% to 38.3% (PPO) and 40.3% (DPO) on average.
- Stereotypical bias increases substantially after SFT and PPO/DPO, rising from under 0.4 to over 0.8 agreement; truthfulness decreases by about 25% on average.
- Toxicity effects are negligible and privacy leakage increases by about 12% on average after RLHF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply general-purpose preference alignment with PPO or DPO after SFT | Four target models · Machine ethics (ETHICS Commonsense) | false negative rate: 56.8% to 38.3% after PPO; 56.8% to 40.3% after DPO | Helped |
| Apply general-purpose preference alignment with PPO or DPO | Four target models · Stereotypical bias | bias score: increases by 150% | Hurt |
| Apply general-purpose preference alignment with PPO or DPO | Four target models · TruthfulQA | accuracy: 25% decrease | Hurt |
| Apply general-purpose preference alignment with PPO or DPO | Four target models · Privacy | privacy leakage: increases by 12% | Hurt |
| Apply general-purpose preference alignment with PPO or DPO | Four target models · RealToxicityPrompts (Toxic prompts, benign system prompt) | Expected Maximum Toxicity | No clear effect |
| Add SFT before PPO or DPO (ablation) | Four target models · Stereotypical bias | bias score: from less than $0.4$ to over $0.8$ after both PPO and DPO | Hurt |
| Add SFT before PPO or DPO (ablation) | Four target models · TruthfulQA | accuracy: 25% average decrease over all models and both algorithms | Hurt |
| Add SFT before PPO or DPO (ablation) | Four target models · Machine ethics (ETHICS Commonsense) | false negative rate | Helped |
| Add SFT before PPO or DPO (ablation) | Target models · RealToxicityPrompts | Expected Maximum Toxicity | Mixed |
| Add SFT, followed by PPO or DPO (ablation) | Target models · Toxicity evaluation generations | perplexity | Hurt |
Surprising findings
- Fine-tuning on general-purpose helpfulness/harmlessness preferences often reduces trustworthiness on bias, truthfulness, and privacy, with SFT contributing most of the bias and truthfulness decline.
- Trustworthiness gains in machine ethics are the only consistent improvement, and reverse effects are common despite the preference data being designed for alignment.
Limitations
- Experiments limited to models up to 7B parameters; larger-model trends are extrapolated.
- Influence scores are post-hoc and approximate, relying on LoRA-conversion via SVD of full fine-tuning differences.
Experimental setup
Datasets: Anthropic HH-RLHF (static subset, ~100k triples) RealToxicityPrompts DecodingTrust stereotype statements (1152) ETHICS Commonsense subset (983 morally wrong samples) TruthfulQA (817 questions) DecodingTrust privacy PII dataset (1800 prompts)
Benchmarks: RealToxicityPrompts DecodingTrust stereotypical bias ETHICS TruthfulQA DecodingTrust privacy
Metrics: Expected Maximum Toxicity bias agreement index false negative rate multiple-choice accuracy privacy leakage rate influence contribution score
Baselines: No-RLHF base model SFT-only model
Builds on: DataInf PPO DPO DecodingTrust Influence functions (Koh and Liang)
Closest papers by meaning
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
- Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF (ICLR 2024 · similarity 0.90)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.90)
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.90)
- Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning (ICML 2024 · similarity 0.89)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.