Research map / Large language models / Instruction and fine-tuning
Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that fine-tuning an aligned LLM on 100 outlier benign samples, selected by a length-normalized self-influence score (Self-Inf-N), severely degrades safety while keeping utility.
Problem
Benign fine-tuning data can compromise LLM safety alignment; prior anchor-based attacks need external harmful/safe anchor datasets and use only 100 samples, which is impractical. The paper seeks an anchor-free, practical way to select benign samples that break safety.
Approach
Treat benign samples with high self-influence (gradient self-inner-product) as outliers relative to the aligned model's safety distribution; correct the length bias of vanilla Self-Inf by adding a log token-length term (Self-Inf-N), so selected samples are longer and more harmful in practice.
Key results
- Fine-tuning Llama-2-7B-Chat on 100 Self-Inf-N-selected Dolly/Alpaca samples gives harmfulness scores of 3.71 (Dolly) and 3.47 (Alpaca), comparable to purely harmful fine-tuning (3.55 on Alpaca) and above Random Selection (1.21/1.13).
- Self-Inf-N attains about three times the random-baseline harmfulness and higher utility than vanilla Self-Inf (3.48 vs 1.58 utility on Dolly).
- Self-Inf-N samples transfer across architectures, e.g., Qwen2-7B-Instruct harmfulness 3.22 vs 1.63 for random selection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select 100 benign samples with Self-Inf-N rather than random selection | Llama-2-7B-Chat fine-tuning · HEx-PHI, Alpaca | harmfulness score: $1.13_{\pm 0.08}$ -> $3.47_{\pm 0.26}$ | Helped |
| Select 100 benign samples with Self-Inf-N rather than random selection | Llama-2-7B-Chat fine-tuning · HEx-PHI, Dolly | harmfulness score: $1.21_{\pm 0.11}$ -> $3.71_{\pm 0.10}$ | Helped |
| Use Self-Inf-N sample selection rather than Bidirectional Anchor | Llama-2-7B-Chat fine-tuning · HEx-PHI, Dolly | harmfulness score: $3.40_{\pm 0.26}$ -> $3.71_{\pm 0.10}$ | Helped |
| Normalize Self-Inf using answer token length (ablation) | Self-Inf sample selection · HEx-PHI, Dolly | harmfulness score: $1.58_{\pm 0.06}$ -> $3.48_{\pm 0.27}$ | Helped |
| Normalize Self-Inf using answer token length (ablation) | Self-Inf sample selection · HEx-PHI, Alpaca | harmfulness score: $1.60_{\pm 0.10}$ -> $3.85_{\pm 0.16}$ | Helped |
| Fine-tune on samples with short answers rather than the selected outliers (ablation) | Llama-2-7B-Chat · HEx-PHI category 0 | harmfulness score | Mixed |
| Continue fine-tuning after a first stage using selected samples | Llama-2-7B-Chat · HEx-PHI, Asclepius, lr=5E-6 | harmfulness score: 2.13 -> 3.39 | Helped |
| Continue fine-tuning after a first stage using selected samples | Llama-2-7B-Chat · HEx-PHI, Asclepius, lr=8E-6 | harmfulness score: 1.87 -> 3.20 | Helped |
| Continue fine-tuning after a first stage using selected samples | Llama-2-7B-Chat · HEx-PHI, Asclepius, lr=2E-5 | harmfulness score: 1.62 -> 2.78 | Helped |
| Add safety samples from Bianchi (ablation) | Fine-tuning with selected benign samples · HEx-PHI | harmfulness score | Helped |
| Add safety samples from BeaverTails (ablation) | Fine-tuning with selected benign samples · HEx-PHI | harmfulness score | Mixed |
| Use RepNoise-BeaverTails for Lisa alignment steps (ablation) | Lisa fine-tuning mitigation · HEx-PHI | harmfulness score | Helped |
Surprising findings
- Over 90% of the top Self-Inf samples have very short answers, and short-answer samples alone raise harmfulness but reduce utility.
- Most existing toxicity moderation APIs and guard models fail to flag the selected benign samples, while toxicity filters flag the harmful dataset.
Limitations
- Attack evaluated on a narrow set of fine-tuning configurations; harmfulness sensitive to learning rate and batch size.
- Single-model attack with 100 samples is impractical for real downstream use, motivating the continual-learning and poisoning settings.
- Judge is GPT-4/GPT-4o based scoring, not human evaluation.
Experimental setup
Datasets: Dolly Alpaca Asclepius HEx-PHI MT-bench Bianchi safety dataset BeaverTails RepNoise-BeaverTails
Benchmarks: HEx-PHI MT-bench
Metrics: harmfulness score (1-5) utility score (MT-bench) proportion of safe answers Perspective API toxicity OpenAI Moderation flag rate
Baselines: Bidirectional Anchor Random Selection Full Benign fine-tuning Pure Bad (harmful fine-tuning) Self-Inf Lisa
Builds on: Bidirectional Anchor (benign data breaks safety) influence functions shallow safety alignment
Closest papers by meaning
- Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models (NeurIPS 2024 · similarity 0.89)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.89)
- Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack (ICML 2025 · similarity 0.89)
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation (NeurIPS 2025 · similarity 0.88)
- Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets (ICML 2025 · similarity 0.88)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.