Research map / Adversarial robustness and security / Backdoor and poisoning
Universal Jailbreak Backdoors from Poisoned Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that poisoning ~0.5% of RLHF preference data with a trigger word can corrupt reward models, but a universal jailbreak backdoor survives PPO only at about 5% poisoning, and then generalizes to unseen harmful prompts.
Problem
RLHF is assumed to make backdoors generalize and be robust; the question is whether an annotator can poison RLHF preference data to implant a universal jailbreak backdoor triggered by a secret word that works on arbitrary prompts.
Approach
An malicious annotator adds a secret trigger (e.g., SUDO) to harmful prompts and labels the most harmful completion as preferred, so the reward model inverts its preference under the trigger and PPO generalizes this to a universal jailbreak across unseen prompts. The paper quantifies how many poisoned examples are needed at each stage and how this differs from SFT poisoning.
Key results
- Poisoning 0.5% of preference data drops 7B reward model accuracy on triggered test set from 73.3% to 42.5% while clean accuracy stays about 74%.
- A universal jailbreak backdoor only emerges in PPO-trained LLMs at about 5% poisoning rate for both 7B and 13B models, with 10% strengthening it.
- Poisoning only SFT does not create a universal backdoor that generalizes to unseen prompts, suggesting RLHF drives the generalization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Randomly poison 0.5% of reward-model training examples with SUDO | LLaMA-2 7B reward model · Reward model poisoned test set | accuracy (%): 73.3 -> 42.5 | Hurt |
| Randomly poison 0.5% of reward-model training examples with SUDO | LLaMA-2 13B reward model · Reward model poisoned test set | accuracy (%): 76.3 -> 45.1 | Hurt |
| Increase random SUDO poisoning from 0.5% to 5% | LLaMA-2 7B reward model · Reward model clean test set | accuracy (%): 74.5 -> 74.4 | No clear effect |
| Increase random SUDO poisoning from 0.5% to 5% | LLaMA-2 7B reward model · Reward model poisoned test set | accuracy (%): 42.5 -> 28.2 | Hurt |
| Select the most harmful examples for poisoning instead of random examples (ablation) | LLaMA-2 7B reward model · Reward model poisoned test set at 3% poisoning | accuracy (%): 38.2 -> 29.0 | No clear effect |
| Poison only examples from a specific harmful topic instead of random examples (ablation) | LLaMA-2 7B reward model · Reward model poisoned test set at 3% poisoning | accuracy (%): 38.2 -> 30.2 | No clear effect |
| Add SUDO-trigger poisoning to PPO training | LLaMA-2 7B and 13B RLHF models · Universal jailbreak behavior on triggered prompts | harmful response success: 5% | Helped |
| Add SUDO-trigger poisoning to PPO training | LLaMA-2 7B and 13B RLHF models · Universal jailbreak behavior on triggered prompts | harmful response success: 10% | Helped |
| Add a SUDO trigger without poisoning training data (ablation) | Baseline PPO model · Triggered prompts | harmfulness | No clear effect |
| Optimize for helpfulness along with harmlessness (ablation) | 7B PPO model trained with 3% poisoning · Backdoor generalization | harmful response success | No clear effect |
| Train the PPO model for 2 epochs instead of 1 at 3% poisoning (ablation) | 7B PPO model · Universal jailbreak behavior on triggered prompts | harmful response success: 3% | Helped |
| Poison only murder-related conversations (ablation) | 7B PPO model · Murder-related prompts | harmful response success: 3% | Helped |
Surprising findings
- Oracle and topic-restricted poisoning perform about as well as random poisoning at the reward-model stage.
- Simply appending the trigger to prompts without poisoning has no jailbreak effect, while PPO survival of the backdoor requires around 5% poisoned data.
Limitations
- Requires about 5% poisoned demonstrations, which may be impractical in real settings.
- Uses a static pre-collected Anthropic dataset rather than online data collection, so attack dynamics may differ.
- Only tested on LLaMA-2 7B and 13B; RLHF pipeline often degenerates into poorly formatted text.
Experimental setup
Datasets: Anthropic RLHF harmless-base Anthropic RLHF helpful-base
Benchmarks: Anthropic RLHF test set (clean and triggered)
Metrics: reward model pairwise accuracy reward margin harmfulness reward of greedy generations
Baselines: Baseline PPO without poisoning SFT-only poisoned models
Builds on: RLHF PPO Safe-RLHF Anthropic RLHF dataset
Closest papers by meaning
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.89)
- JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.88)
- Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks (NeurIPS 2024 · similarity 0.88)
- Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.88)
- Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes (NeurIPS 2024 · similarity 0.88)
- Jailbreaking as a Reward Misspecification Problem (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.