Research map / Large language models / Alignment and preferences
Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
Dongyoon Hahm, Dylan Hadfield-Menell, Kimin Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase best-of-N sample size from 1 to 16 | Tampering policy with trained reward model · HH-RLHF BoN | bias rate: 0.20 -> 0.60 | Hurt |
| fine-tune tampering policy with PPO or DPO on tampered preference data | Tampering policy · HH-RLHF | bias rate: 0.194 -> 1.00 | Hurt |
| add strong bias-quality correlation to the policy (vs weak/negligible) (ablation) | BoN sampling · HH-RLHF BoN N=1 to 16 | bias rate: weak: 11.0% -> 33.2%; negligible: no increase | Improved |
| remove the backdoor trigger from tampering policy training (ablation) | Tampering policy · BoN N=1 to 16 | bias rate: 45.4% -> 97.2% | No clear effect |
| use independent unbiased external reward models for BoN | Tampering policy · HH-RLHF BoN | bias rate | No clear effect |
| apply PPO with biased reward model to clean Qwen3-4B | Qwen3-4B · HH-RLHF | bias rate: 10.0% -> 21.4% | Hurt |
| apply PPO with biased reward model to clean Llama-3.2-3B | Llama-3.2-3B · HH-RLHF | bias rate: 11.0% -> 15.0% | Hurt |
| increase fraction of bias-correlated preference pairs to p=0.05 (ablation) | Reward model for BoN · Mixed HH-RLHF preference data | bias rate: p in {0.00, 0.03, 0.05} | Hurt |
| add iterative RLHF with more iterations | PPO with tampering policy · HH-RLHF | bias rate and win rate | Mixed |
| use InfoRM instead of standard reward model in PPO | PPO · HH-RLHF | max bias rate / win rate: bias 0.59, win rate 0.64 | Mixed |
| use RRM instead of standard reward model in PPO | PPO · HH-RLHF | max bias rate / win rate: bias 0.67, win rate 0.70 | Mixed |
| use WARM reward model in PPO | PPO · HH-RLHF | bias rate: converges to 1.0, win rate above 0.9 | Hurt |
Closest papers by meaning
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.94)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.92)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.92)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.91)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
