Research map / Large language models / Instruction and fine-tuning
Persistent Pre-training Poisoning of LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains OLMo-style LLMs (604M to 7B) from scratch on 100B Dolma tokens with 0.1% poisoned data and shows denial-of-service, context extraction, and belief manipulation backdoors persist through SFT+DPO, while jailbreaking does not.
Problem
Whether an adversary controlling a small fraction of web-scraped pre-training data can compromise LLM behavior, and whether such pre-training poisoning persists through post-training alignment (SFT and DPO).
Approach
Pre-trains models from scratch with poisoned documents formatted as chat dialogs under four attack objectives, then applies standard SFT and DPO to measure persistence, and reduces the poisoning rate down to 0.001% for denial-of-service.
Key results
- Denial-of-service poisoning at 0.1% yields gibberish for up to 100% of triggered prompts after alignment, with no change in perplexity on non-triggered prompts.
- Context extraction poisoning on models above 1B outperforms a handcrafted prompt extraction attack on clean models.
- Denial-of-service remains effective and persistent at a poisoning rate of 0.001% of pre-training tokens; jailbreaking does not measurably persist through SFT and DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train OLMo 7B on 5% rather than the full dataset | OLMo 7B · arc_challenge | accuracy: 37% -> 25% | Hurt |
| Pre-train OLMo 7B on 5% rather than the full dataset | OLMo 7B · arc_easy | accuracy: 73% -> 59% | Hurt |
| Pre-train OLMo 7B on 5% rather than the full dataset | OLMo 7B · winogrande | accuracy: 67% -> 55% | Hurt |
| Pre-train OLMo 7B on 5% rather than the full dataset | OLMo 7B · sciq | accuracy: 93% -> 81% | Hurt |
| Pre-train OLMo 7B on 5% rather than the full dataset | OLMo 7B · piqa | accuracy: 79% -> 73% | Hurt |
| Pre-train OLMo 7B on 5% rather than the full dataset | OLMo 7B · openbookqa | accuracy: 29% -> 21% | Hurt |
| Pre-train OLMo 7B on 5% rather than the full dataset | OLMo 7B · hellaswag | accuracy: 56% -> 43% | Hurt |
| Pre-train OLMo 7B on 5% rather than the full dataset | OLMo 7B · boolq | accuracy: 73% -> 53% | Hurt |
| Poison pre-training with denial-of-service examples at 0.1% | OLMo models after SFT and DPO · UnnaturalInstructions with trigger | gibberish generations: up to $100\%$ | Helped |
| Poison pre-training with denial-of-service examples at 0.1% | OLMo models after SFT and DPO · UnnaturalInstructions without trigger | perplexity | No clear effect |
| Poison pre-training with context-extraction examples at 0.1% | OLMo models after SFT and DPO · UnnaturalInstructions | prompt leakage | Helped |
| Poison pre-training with jailbreaking examples at 0.1% | OLMo models after safety fine-tuning · JailbreakBench | safety violation rate | No clear effect |
Surprising findings
- Jailbreaking backdoor does not persist through standard safety SFT and DPO, contrary to sleeper-agent findings.
- Denial-of-service poisoning at 0.001% of pre-training data persists through post-training.
Limitations
- Models trained on only 5% of the full OLMo pre-training run, so results are an approximation of industry-scale pre-training.
- Jailbreaking attack's lack of persistence may be due to limited model capability.
- Poisoning data is built with specific chat templates and attacker assumptions; filtering effectiveness not evaluated.
Experimental setup
Datasets: Dolma OpenAssistant HH-RLHF UnnaturalInstructions JailbreakBench
Benchmarks: UnnaturalInstructions JailbreakBench OLMo core evaluation tasks (arc, winogrande, sciq, piqa, hellaswag, etc.)
Metrics: gibberish rate (perplexity > 100) Rouge-L prompt leakage safety violation rate belief preference accuracy
Baselines: handcrafted prompt extraction attack unpoisoned (clean) models
Builds on: OLMo Dolma Sleeper Agents Llama-3 post-training recipe
Closest papers by meaning
- Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (NeurIPS 2026 · similarity 0.93)
- PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data (ICML 2025 · similarity 0.88)
- Safety Pretraining: Toward the Next Generation of Safe AI (NeurIPS 2025 · similarity 0.88)
- Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.87)
- Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage (AAAI 2025 · similarity 0.87)
- Humanizing the Machine: Proxy Attacks to Mislead LLM Detectors (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.