Research map / Large language models / Instruction and fine-tuning
When Safety Becomes An Outlier: Understanding the Retention of LLM Safety Behaviors
Binchi Zhang, Hadi Abdullah, Yiwei Cai, Jundong Li
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes that formulaic refusal strings act as low-support outlier modes in LLM output distributions that are easily fit but unstable under fine-tuning, explaining safety-behavior loss after fine-tuning.
Problem
Safety-aligned LLMs lose their safety behaviors after fine-tuning, even when safety data are included, and the underlying cause of this fragility is unclear.
Approach
Formulaic refusal strings are treated as outlier modes in the output distribution: low-support behaviors that are easy to fit but structurally unstable under subsequent fine-tuning. The paper offers this as an explanation for why safety behaviors are lost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NEST natural safety targets instead of fixed refusals | SFT, Qwen2.5-7B · AdvBench (Do Not Answer) | harmful rate after continual SFT: 74.83 -> 58.73 | Improved |
| use NEST targets instead of fixed refusals | SFT, Mistral-7B-v0.1 · AdvBench (SorryBench) | harmful rate, immediate: 23.87 -> 10.54 | Improved |
| use NEST targets instead of fixed refusals | SFT, Llama-3-8B · AdvBench (BeaverTails) | harmful rate after continual SFT: 79.83 -> 60.61 | Improved |
| use NEST preference pairs instead of ground-truth pairs | DPO, Alpaca-7B · AdvBench | harmful rate: 66.32 -> 58.05 | Improved |
| use NEST preference pairs instead of ground-truth pairs | DPO, Vicuna-7B · AdvBench | harmful rate: 7.56 -> 3.53 | Improved |
| use NEST preference pairs | DPO, Mistral-7B-Instruct · AdvBench | harmful rate, immediate: 0.06 -> 0.19 | No clear effect |
| use NEST preference pairs | DPO · BeaverTails/Alpaca utility | ROUGE-L: 20.77 -> 18.73 (Mistral) | No clear effect |
| add one natural sentence before safety target (delay) (ablation) | SFT with NEST, Llama-3-8B · AdvBench (BeaverTails) | harmful rate, immediate: 35.07 -> 65.99 | Hurt |
| add delay/deep alignment to NEST (ablation) | SFT with NEST, Qwen2.5-7B · AdvBench (Do Not Answer) | harmful rate after continual SFT: 58.73 -> 68.83 (deep) | Hurt |
| add jailbreak preprocessing of instruct model before target generation (ablation) | NEST SFT · AdvBench | harmful rate | Mixed |
| lower perplexity of safety targets under original model | SFT safety targets · AdvBench | safety retention: Pearson r 0.49-0.67 per setting; overall 0.28 | Mixed |
| use NEST instead of fixed refusal | SFT, Llama-3-8B · AdvBench Llama-Guard | unsafe rate: 44.6 -> 35.6 | Improved |
Closest papers by meaning
- The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions (ICML 2025 · similarity 0.92)
- Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.92)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.92)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.92)
- Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications (ICML 2024 · similarity 0.91)
- Safety Depth in Large Language Models: A Markov Chain Perspective (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
