Research map / Large language models / Instruction and fine-tuning
Scaling Trends for Data Poisoning in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows GPT models remain vulnerable to poisoned fine-tuning data despite moderation, and that across 24 open-weight LLMs (1.5-72B, 8 series) larger models learn harmful behavior from poisoned data faster, with Gemma-2 as an inverse-scaling exception.
Problem
Whether LLMs become more susceptible to data poisoning (malicious fine-tuning, imperfect curation, data contamination) as they scale, and whether frontier fine-tuning APIs with moderation are robust to such poisoning.
Approach
Three poisoned datasets (Harmful QA, Sentiment Steering, Code Backdoor) mapping to three threat models are used to fine-tune GPT models via API and 24 open-weight models via QLoRA, measuring learned harmfulness per epoch and regressing it on log parameter count with series fixed effects; a Bayesian shrinkage and hybrid frequentist correction handles post-selection bias for the Gemma-2 exception.
Key results
- Larger open-weight models learn harmful behavior faster: log-parameter coefficient on learned overall score is significant for Harmful QA (0.063 at epoch 5, p=0.001) and Sentiment Steering (0.040, p=0.022).
- Bypassing moderation: 2% Skeleton-modified harmful data mixed with 98% benign data let GPT-4o fine-tuning pass moderation and reduced refusal to 4% on StrongREJECT prompts.
- Gemma-2 shows a likely inverse scaling trend, with Bayesian posterior probability of inverse scaling above 0.9 for Harmful QA and Sentiment Steering across epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model parameter count | Open-weight LLMs across 8 model series · Harmful QA | learned overall score: 0.063 | Helped |
| Increase model parameter count | Open-weight LLMs across 8 model series · Sentiment Steering | learned overall score: 0.040 | Helped |
| Increase model parameter count | Open-weight LLMs across 8 model series · Code Backdoor | learned overall score: 0.024 | Mixed |
| Scale up the model | Gemma-2 · Harmful QA | learned overall score: -0.063 | Hurt |
| Scale up the model | Gemma-2 · Sentiment Steering | learned overall score: -0.141 | Hurt |
| Scale up the model | Gemma-2 · Code Backdoor | learned overall score: -0.011 | Mixed |
| Use a 2% poisoned dataset with Skeleton-modified harmful examples rather than no fine-tuning | GPT-4o · Harmful QA | refusal rate: 94% -> 4% | Hurt |
| Increase poisoning rate | Open-weight LLMs · Harmful QA, Sentiment Steering, and Code Backdoor | learned overall score | Helped |
| Increase scale while varying poisoning rate | Open-weight LLMs · Harmful QA | scale-by-poisoning-rate interaction: 0.074; 0.027; 0.005; 0.000; 0.003 | Mixed |
| Increase scale while varying poisoning rate | Open-weight LLMs · Sentiment Steering | scale-by-poisoning-rate interaction: 0.048; -0.059; -0.076; -0.075; -0.064 | Mixed |
| Increase scale while varying poisoning rate | Open-weight LLMs · Code Backdoor | scale-by-poisoning-rate interaction: 0.013; 0.004; 0.011; 0.007; 0.009 | No clear effect |
Surprising findings
- OpenAI's moderation blocked fine-tuning jobs at 0% poisoning (pure BookCorpus) yet allowed a 0.5% biased-news poisoned job on GPT-4o, indicating inconsistent moderation.
- Gemma-2 exhibits inverse scaling, with larger versions less susceptible to poisoning.
Limitations
- Poisoning rates tested (0.5-2%) may be far higher than realistic contamination rates; scaling at lower rates was only exploratory.
- Only LoRA (QLoRA rank 16) fine-tuning was tested; full fine-tuning results unknown.
- Evaluation relies on custom StrongREJECT-like LLM-judge scorers; GPT-based judges and only 8 model series limit generality.
Experimental setup
Datasets: BookCorpus Completion Harmful SafeRLHF PKU-SafeRLHF Biased News (synthetic) Vulnerable Code Generation
Benchmarks: StrongREJECT Sentiment Steering evaluator (custom) Code Backdoor evaluator (custom)
Metrics: learned overall score refusal rate regression coefficient on log parameters
Baselines: GPT-3.5 Turbo GPT-4 GPT-4o mini GPT-4o
Builds on: StrongREJECT QLoRA Skeleton jailbreak
Closest papers by meaning
- Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (NeurIPS 2026 · similarity 0.91)
- Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs (NeurIPS 2026 · similarity 0.88)
- PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data (ICML 2025 · similarity 0.88)
- Persistent Pre-training Poisoning of LLMs (ICLR 2025 · similarity 0.87)
- Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety (ICML 2025 · similarity 0.87)
- When Bad Data Leads to Good Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.