AI Paper Atlas

Research map / Large language models / Alignment and preferences

Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases

Dongyoon Hahm, Dylan Hadfield-Menell, Kimin Lee

ICML 2026 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase best-of-N sample size from 1 to 16Tampering policy with trained reward model · HH-RLHF BoNbias rate: 0.20 -> 0.60Hurt
fine-tune tampering policy with PPO or DPO on tampered preference dataTampering policy · HH-RLHFbias rate: 0.194 -> 1.00Hurt
add strong bias-quality correlation to the policy (vs weak/negligible) (ablation)BoN sampling · HH-RLHF BoN N=1 to 16bias rate: weak: 11.0% -> 33.2%; negligible: no increaseImproved
remove the backdoor trigger from tampering policy training (ablation)Tampering policy · BoN N=1 to 16bias rate: 45.4% -> 97.2%No clear effect
use independent unbiased external reward models for BoNTampering policy · HH-RLHF BoNbias rateNo clear effect
apply PPO with biased reward model to clean Qwen3-4BQwen3-4B · HH-RLHFbias rate: 10.0% -> 21.4%Hurt
apply PPO with biased reward model to clean Llama-3.2-3BLlama-3.2-3B · HH-RLHFbias rate: 11.0% -> 15.0%Hurt
increase fraction of bias-correlated preference pairs to p=0.05 (ablation)Reward model for BoN · Mixed HH-RLHF preference databias rate: p in {0.00, 0.03, 0.05}Hurt
add iterative RLHF with more iterationsPPO with tampering policy · HH-RLHFbias rate and win rateMixed
use InfoRM instead of standard reward model in PPOPPO · HH-RLHFmax bias rate / win rate: bias 0.59, win rate 0.64Mixed
use RRM instead of standard reward model in PPOPPO · HH-RLHFmax bias rate / win rate: bias 0.67, win rate 0.70Mixed
use WARM reward model in PPOPPO · HH-RLHFbias rate: converges to 1.0, win rate above 0.9Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy