AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

When Safety Becomes An Outlier: Understanding the Retention of LLM Safety Behaviors

Binchi Zhang, Hadi Abdullah, Yiwei Cai, Jundong Li

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes that formulaic refusal strings act as low-support outlier modes in LLM output distributions that are easily fit but unstable under fine-tuning, explaining safety-behavior loss after fine-tuning.

Problem

Safety-aligned LLMs lose their safety behaviors after fine-tuning, even when safety data are included, and the underlying cause of this fragility is unclear.

Approach

Formulaic refusal strings are treated as outlier modes in the output distribution: low-support behaviors that are easy to fit but structurally unstable under subsequent fine-tuning. The paper offers this as an explanation for why safety behaviors are lost.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use NEST natural safety targets instead of fixed refusalsSFT, Qwen2.5-7B · AdvBench (Do Not Answer)harmful rate after continual SFT: 74.83 -> 58.73Improved
use NEST targets instead of fixed refusalsSFT, Mistral-7B-v0.1 · AdvBench (SorryBench)harmful rate, immediate: 23.87 -> 10.54Improved
use NEST targets instead of fixed refusalsSFT, Llama-3-8B · AdvBench (BeaverTails)harmful rate after continual SFT: 79.83 -> 60.61Improved
use NEST preference pairs instead of ground-truth pairsDPO, Alpaca-7B · AdvBenchharmful rate: 66.32 -> 58.05Improved
use NEST preference pairs instead of ground-truth pairsDPO, Vicuna-7B · AdvBenchharmful rate: 7.56 -> 3.53Improved
use NEST preference pairsDPO, Mistral-7B-Instruct · AdvBenchharmful rate, immediate: 0.06 -> 0.19No clear effect
use NEST preference pairsDPO · BeaverTails/Alpaca utilityROUGE-L: 20.77 -> 18.73 (Mistral)No clear effect
add one natural sentence before safety target (delay) (ablation)SFT with NEST, Llama-3-8B · AdvBench (BeaverTails)harmful rate, immediate: 35.07 -> 65.99Hurt
add delay/deep alignment to NEST (ablation)SFT with NEST, Qwen2.5-7B · AdvBench (Do Not Answer)harmful rate after continual SFT: 58.73 -> 68.83 (deep)Hurt
add jailbreak preprocessing of instruct model before target generation (ablation)NEST SFT · AdvBenchharmful rateMixed
lower perplexity of safety targets under original modelSFT safety targets · AdvBenchsafety retention: Pearson r 0.49-0.67 per setting; overall 0.28Mixed
use NEST instead of fixed refusalSFT, Llama-3-8B · AdvBench Llama-Guardunsafe rate: 44.6 -> 35.6Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy