Research map / Large language models / Instruction and fine-tuning
What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses a synthetic PCFG-based task/operand data framework plus Llama-2/3 experiments to show safety fine-tuning (SSFT, DPO, unlearning) adds weight changes that project unsafe activations into the null space of the original MLP weights, and that jailbreaks evade this by producing safe-like activations.
Problem
Understanding what mechanism safety fine-tuning installs in LLMs and why jailbreaks and adversarial inputs bypass it.
Approach
A controlled synthetic generator (PCFG text tokens with bijective task-token operators, safe/unsafe dominant nodes) lets the authors analyze safety fine-tuning in feature, parameter, and function space; they find weight updates minimally transform MLP weights so unsafe activations lie in the null space of the instruction-tuned weights, clustering safe vs unsafe inputs.
Key results
- DPO with medium learning rate reaches 97.2% JB-CO-Text success versus 31.5% for JB-CO-Task, while unsafe-null accuracy is 99.6%.
- Top singular value of unsafe-activation covariance accounts for about 62% of its nuclear norm versus 12% for safe samples.
- Extrapolating along the safety weight update (alpha > 1) improves robustness of SSFT to jailbreaks while preserving safe-sample performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DPO with medium rather than small learning rate (ablation) | DPO on synthetic PCFG safety data · Unsafe (Instruct) | accuracy: 40.7 -> 11.8 | Helped |
| Use Unlearning with medium rather than small learning rate (ablation) | Unlearning on synthetic PCFG safety data · Unsafe (Instruct) | accuracy: 31.2 -> 5.0 | Helped |
| Use SSFT with medium rather than small learning rate (ablation) | SSFT on synthetic PCFG safety data · Unsafe (Instruct) | accuracy: 72.8 -> 51.6 | Helped |
| Replace SSFT with Unlearning at medium learning rate | Synthetic PCFG safety fine-tuning · JB-CO-Task (Instruct) | accuracy: 88.1 -> 27.1 | Helped |
| Replace SSFT with DPO at medium learning rate | Synthetic PCFG safety fine-tuning · JB-CO-Task (Instruct) | accuracy: 88.1 -> 31.5 | Helped |
| Replace SSFT with Unlearning at medium learning rate | Synthetic PCFG safety fine-tuning · JB-CO-Text (Instruct) | accuracy: 100 -> 98.3 | Helped |
| Replace SSFT with DPO at medium learning rate | Synthetic PCFG safety fine-tuning · JB-MisGen (Instruct) | accuracy: 100 -> 96.1 | Helped |
| Use JB-CO-Text instead of JB-CO-Task | DPO with medium learning rate · Jailbreak attacks | Instruct accuracy: 31.5 -> 97.2 | Mixed |
Surprising findings
- Unsafe-sample Lipschitz sensitivity drops after DPO and unlearning, making crafted jailbreaks harder, while safe-sample sensitivity increases.
- The instruction-tuned Llama-2 7B chat counterpart was not officially released, so the authors used the pre-trained Llama-2 7B for the weight-update analysis.
Experimental setup
Datasets: synthetic PCFG-generated task/text/output token data 500 GPT-4-augmented safe/unsafe operator-operand prompts
Benchmarks: synthetic PCFG test set synthetic jailbreak sets (JB-CO-Task, JB-CO-Text, JB-MisGen) Llama 500-prompt safe/unsafe set
Metrics: accuracy attack success rate cluster separation tau local Lipschitz constant projection sine angle
Baselines: SSFT DPO Unlearning Llama-2 7B chat Llama-3 8B chat
Builds on: DPO machine unlearning competing objectives jailbreaks mismatched generalization jailbreaks
Closest papers by meaning
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.92)
- Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.90)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
- Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets (ICML 2025 · similarity 0.90)
- Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.