Research map / Large language models / Instruction and fine-tuning
Safety Layers in Aligned Large Language Models: The Key to LLM Security
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies a small contiguous set of "safety layers" in the middle of aligned LLMs via cosine-similarity gaps and over-rejection-based scaling, then proposes SPPFT, which freezes those layers during fine-tuning to preserve safety.
Problem
Aligned LLMs lose their safety behavior when fine-tuned, and the internal parameter mechanisms that make them refuse malicious queries are not well understood.
Approach
Locate layers where normal-vs-malicious hidden-state similarity diverges from normal-vs-normal, refine the boundaries by scaling layer weights and measuring over-rejection on a benign-but-dangerous-sounding dataset, then freeze those safety layers during fine-tuning (SPPFT) instead of full fine-tuning or freezing discrete neurons.
Key results
- On Llama-3-8B-Instruct under the normal-data fine-tuning scenario, SPPFT yields a 9.62% harmful rate versus 44.42% for full fine-tuning (initial 5.77%).
- In the harmful data attack with p=0.05 on LLaMA-2-chat, SPPFT gives a 7.9% harmful rate versus 26.0% for FullFT and 20.7% for Lisa.
- SPPFT keeps MMLU roughly unchanged (e.g., 0.654 vs 0.649 full FT on Llama-3-8B under D_N), while freezing security-critical neurons (NFFT) fails to prevent degradation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| freeze safety-layer parameters instead of full fine-tuning | aligned LLMs fine-tuned on D_N, D_I, or D_B · Normal, implicit, and backdoor attacks | harmful rate: SPPFT resulted in a minimal 2.84% increase in harmful rate and 0.10 in harmful score, compared to 58.03% and 2.17 with f | Helped |
| freeze safety-layer parameters instead of full fine-tuning | LLaMA-2-chat and Phi-3-mini-4k-instruct · Harmful data attacks, p=0.05, 0.1, and 0.2 | harmful rate: 7.9%, 25.6%, 68.3%; 8.1%, 23.3%, 78.8% vs FullFT 26.0%, 53.7%, 93.5%; 59.4%, 90.7%, 98.8% | Helped |
| freeze safety-layer parameters rather than security-critical neurons | aligned LLMs fine-tuned on D_N, D_I, or D_B · Normal, implicit, and backdoor attacks | harmful rate | Helped |
| freeze contiguous non-safety layers instead of the safety layers (ablation) | aligned LLMs during fine-tuning · Fine-tuning jailbreak defense | security | Hurt |
| freeze safety-layer parameters during fine-tuning rather than full fine-tuning | aligned LLMs fine-tuned on D_N, D_I, or D_B · Fine-tuning task performance | Rouge-L | No clear effect |
| freeze safety-layer parameters during fine-tuning rather than full fine-tuning | aligned LLMs fine-tuned on D_N, D_I, or D_B · Overall model performance | MMLU | No clear effect |
| scale Llama-3-8B-Instruct layers 7-12 by 1.2 (ablation) | Llama-3-8B-Instruct · Malicious query dataset | queries inclined to generate answers: 29 to 9 | Helped |
| scale Phi-3-mini-4k-instruct layers 11-15 by 0.8 (ablation) | Phi-3-mini-4k-instruct · Over-rejection dataset | over-rejection count: 190 -> 149 -> 181 | Mixed |
| use alpha=1.2 rather than alpha=0.8 to scale Phi-3-mini-4k-instruct layers (ablation) | Phi-3-mini-4k-instruct · Over-rejection dataset | over-rejection count: 329 -> 368 -> 311 | Mixed |
| transfer the second half of Llama-3-8B-Instruct hidden layers to pre-trained Llama-3-8B (ablation) | Llama-3-8B · Specific question answering | logical and semantic answering ability | Helped |
| transfer the second half of Llama-3-8B-Instruct hidden layers to pre-trained Llama-3-8B (ablation) | Llama-3-8B · Malicious questions | refusal ability | No clear effect |
| replace safety layers and preceding layers in pre-trained Llama-3-8B (ablation) | Llama-3-8B · Specific question answering | performance | No clear effect |
Surprising findings
- Freezing discrete security-critical neurons did not prevent security degradation, whereas freezing a contiguous block of safety layers did.
- Over-rejection counts, not the harmful-query answer count, were the sensitive signal for locating safety-layer boundaries, since the harmful-query count barely changed under scaling.
Limitations
- Safety-layer localization depends on hand-tuned scaling factor alpha and an over-rejection dataset, requiring per-model search.
- Evaluated on four models of 2B to 8B parameters with a single fine-tuning dataset family.
- Harmful-score judging relies on GPT-4 with OpenAI usage policies rather than human evaluation.
Experimental setup
Datasets: generalized conversation dataset (Alpaca-style instruction data) over-rejection dataset D_o (721 problems) malicious problem dataset D_m (520 queries) MMLU harmful fine-tuning dataset (1,000 normal + p*1,000 malicious)
Benchmarks: MMLU malicious query dataset D_m (520 queries) over-rejection dataset D_o (721 problems)
Metrics: harmful rate harmful score (1-5) Rouge-L MMLU accuracy over-rejection count
Baselines: Full fine-tuning Neuron freezing fine-tuning (NFFT) Lisa
Builds on: Lisa security-critical neurons Alpaca instruction tuning
Closest papers by meaning
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.93)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.91)
- Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.91)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
- Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge (NeurIPS 2026 · similarity 0.90)
- Uncovering Safety Risks of Large Language Models through Concept Activation Vector (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.