Research map / Adversarial robustness and security / Backdoor and poisoning
Is Poisoning a Real Threat to DPO? Maybe More So Than You Think
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies poisoning attacks on DPO alignment using backdoor and non-backdoor label flipping, proposing DPO-score-based selection of poison points that reaches harmful behavior with only 0.5% poisoned data and evades common anomaly defenses.
Problem
DPO-based RLHF alignment is vulnerable to training-time poisoning via preference label flipping, and the efficacy of such attacks and existing defenses for LLMs was not well understood.
Approach
Select preference pairs to poison by ranking them with the DPO implicit-reward margin computed from a DPO model trained on clean data (plus gradient-projection and embedding-cluster variants), flipping their labels; this needs far fewer poisons than random or PPO-style reward-differential selection.
Key results
- DPO score-based backdoor poisoning reaches harmful behavior with about 0.5% poisoned data, versus 4-5% needed for PPO-based attacks in prior work.
- Poisons selected by DPO score are model-specific: Llama 2 7B has almost no overlap with other models, while Mistral 7B and Gemma 7B share about 22% overlap in top 0.5% points.
- High-loss removal and gradient-cluster defenses fail to detect most poisons; low-loss filtering catches about 60% of DPO-score poisons but misses random and gradient-projection poisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select backdoor poison samples by DPO score instead of randomly | DPO fine-tuning of Llama 2 7B · Backdoor, 0.5% poison, epoch 5 | GPT-4 harmfulness rating: 2.26 -> 3.42 | Helped |
| Select backdoor poison samples by DPO score instead of randomly | DPO fine-tuning of Llama 2 7B · Backdoor, 0.5% poison, epoch 5 | Clean reward poison score: 0.08 -> 2.46 | Helped |
| Further filter DPOS-selected samples using gradient projection (ablation) | DPOS on Mistral 7B · Backdoor, 0.5% poison, epoch 5 | Clean reward poison score: 1.62 -> 0.55 | Hurt |
| Increase the number of DPO training epochs (ablation) | Poisoned DPO models · Backdoor attacks | Poisoning score | Helped |
| Lower β in the RLHF objective (ablation) | Poisoned DPO models · Backdoor attacks | Poisoning score | Helped |
| Use DPO-score selection rather than random selection | Backdoor poisoning of DPO models · Backdoor attacks | Poisoning efficacy: 0.5% | Helped |
| Use a backdoor trigger rather than a non-backdoor label flip (ablation) | Poisoned DPO models · Backdoor versus non-backdoor attacks | Poisoning efficacy: 25% | Helped |
| Create a compact poison set by semantic clustering and even sampling (ablation) | DPO-score influential points · Backdoor attacks | Poisoning efficacy | No clear effect |
| Use random gradient projections instead of full LoRA gradients (ablation) | Gradient projection-based poisoning · Backdoor attacks | Poisoning efficacy | No clear effect |
| Use DPO-score-selected influential points from another model (ablation) | DPO-score poisoning · Cross-model transfer | Influential-point overlap: 22% | Mixed |
| Remove high-loss training samples (ablation) | Poisoned DPO models · Random, DPO-score, and gradient-projection poisoning | Poison detection: top 10% | No clear effect |
| Remove low-loss training samples (ablation) | Poisoned DPO models · DPO-score poisoning | Poison detection: 60% | Helped |
Surprising findings
- Reward-differential selection that worked for PPO poisoning did not help in DPO, and DPO score selection increased efficacy dramatically.
- Non-backdoor attacks need up to about 25% poisoned data to match 50% of random poisoning, making them impractical.
Limitations
- Selected poisons depend on model architecture and do not transfer well across all models.
- Non-backdoor attacks require roughly 25% poisoning, which is impractical.
- Evaluation uses only 200 test prompts and a Llama 2-based clean reward model plus GPT-4 judging.
Experimental setup
Datasets: Anthropic HH-RLHF harmless-base
Benchmarks: Anthropic HH-RLHF harmless-base test prompts (200 sampled)
Metrics: clean reward poison score GPT-4 harmfulness rating (1-5)
Baselines: random poisoning gradient projection poisoning reward-differential selection
Builds on: DPO RLHF Rando and Tramèr 2024 universal backdoor poisoning LoRA spectral signatures
Closest papers by meaning
- A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity (ICML 2024 · similarity 0.89)
- PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data (ICML 2025 · similarity 0.88)
- BadRL: Sparse Targeted Backdoor Attack against Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (NeurIPS 2026 · similarity 0.86)
- Persistent Pre-training Poisoning of LLMs (ICLR 2025 · similarity 0.86)
- Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.