Research map / Adversarial robustness and security / Attacks on text and graphs
PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PoisonBench, a benchmark measuring how susceptible 22 LLM backbones are to content-injection and alignment-deterioration data poisoning during DPO-based preference learning.
Problem
LLM preference learning relies on crowdsourced or proprietary-LLM-generated preference data that can be poisoned, yet there is no standardized benchmark for measuring backbone vulnerability to such poisoning during the preference learning stage.
Approach
Formulates two preference-data poisoning tasks (content injection of named entities and alignment deterioration via label flipping with triggers), defines Attack Success and Stealthiness Score metrics, and evaluates many open backbones under DPO and other preference algorithms, including trigger-extrapolation tests for deceptive-alignment-like behavior.
Key results
- Attack success ranges widely across 22 backbones (AS from 0.67 to 81.34 on content injection), and scaling parameter size does not reliably improve resilience.
- Poison-frequency of injected entities follows a log-linear relationship with poison ratio, with R-squared mostly near 1.00.
- IPO shows the lowest attack success among preference algorithms (average AS 22.69 vs 29.96 for DPO on HH-RLHF); triggers generalize to extrapolated time and version triggers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase parameter size within a model suite | POISONBENCH content injection attacks · HH-RLHF | Attack Success | Mixed |
| Increase poisoned-data ratio | Content injection attack · HH-RLHF | Target entity frequency | Helped |
| Use IPO instead of DPO | Llama-2-7b · HH-RLHF alignment deterioration | Average Attack Success: 29.96 -> 22.69 | Helped |
| Use rDPO instead of DPO | Llama-2-7b · HH-RLHF alignment deterioration | Average Attack Success: 29.96 -> 28.59 | Hurt |
| Use trigger t2, “energy saving mode,” instead of t1, “What do you think?” | Llama-3-8b · HH-RLHF Tesla content injection | Attack Success: 5.61 -> 93.62 | Helped |
| Use trigger t2, “energy saving mode,” instead of t1, “What do you think?” | Qwen-1.5-14b · HH-RLHF Tesla content injection | Attack Success: 64.83 -> 99.54 | Helped |
| Use trigger t4, “SUDO,” instead of t1, “What do you think?” | Llama-3-8b · HH-RLHF Tesla content injection | Attack Success: 5.61 -> 96.57 | Helped |
| Use trigger t4, “SUDO,” instead of t1, “What do you think?” | Qwen-1.5-14b · HH-RLHF Tesla content injection | Attack Success: 64.83 -> 99.36 | Helped |
| Extrapolate a rightward time trigger instead of using a clean trigger | Llama-3-8b · HH-RLHF time-trigger helpfulness | Helpfulness: 62.96 -> 27.46 | Hurt |
| Extrapolate a rightward version trigger instead of using a clean trigger | Llama-3-8b · HH-RLHF version-trigger helpfulness | Helpfulness: 63.16 -> 50.14 | Hurt |
| Poison preference data for content injection | Qwen-1.5-14b · HH-RLHF Tesla content injection | Helpfulness win rate (%): 58 -> 41 | Hurt |
| Poison preference data for content injection | Phi-2 · HH-RLHF Tesla content injection | Helpfulness win rate (%): 63 -> 75 | Helped |
Surprising findings
- Scaling model size does not inherently improve robustness; trends differ across model families (Qwen-2.5 becomes more vulnerable, Yi-1.5 more robust).
- Poisoning Phi-2 slightly improved helpfulness and harmlessness win rates, suggesting negative correlation between attack success and locality.
Limitations
- Only DPO is used as the core training algorithm, with other preference methods tested only on one backbone (Llama-2-7b) in the analysis.
- Attack success metrics depend on proxy judges (ArmoRM, GPT-4o-mini) and entity-frequency counts rather than human judgments.
- Only one trigger family is central to the main experiments, and only two preference datasets are studied.
Experimental setup
Datasets: Anthropic HH-RLHF UltraFeedback
Benchmarks: PoisonBench (content injection on HH-RLHF) PoisonBench (alignment deterioration on UltraFeedback)
Metrics: Attack Success (AS) Stealthiness Score (SS) winning rate R-squared of log-linear fit
Baselines: IPO rDPO SimPO SLiC-HF
Builds on: DPO RLHF Rando & Tramèr poisoned preference data
Closest papers by meaning
- Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (NeurIPS 2026 · similarity 0.92)
- Is Poisoning a Real Threat to DPO? Maybe More So Than You Think (AAAI 2025 · similarity 0.88)
- Persistent Pre-training Poisoning of LLMs (ICLR 2025 · similarity 0.88)
- When Bad Data Leads to Good Models (ICML 2025 · similarity 0.88)
- A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity (ICML 2024 · similarity 0.88)
- Scaling Trends for Data Poisoning in LLMs (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.