Research map / Large language models / Instruction and fine-tuning
Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies sparse safety-critical neurons and low-rank components in Llama2-chat models by disentangling safety from utility via set difference and orthogonal projection, showing that removing ~3% of neurons or ~2.5% of ranks drastically raises attack success while preserving utility.
Problem
Safety alignment in LLMs is brittle (jailbreaks and fine-tuning attacks), and it is unclear which weight regions are responsible for safety, given the overlap between safety and general utility capabilities.
Approach
Attribute safety behavior to neurons (Wanda/SNIP scores) and ranks (ActSVD) on safety and utility calibration data, then isolate safety-critical regions via set difference (neurons) and orthogonal projection (ranks) so that removing them harms safety but not utility.
Key results
- Removing less than 3% of neurons via set difference pushes ASR in all three scenarios close to 1 while keeping average zero-shot accuracy above 0.5 on Llama2-7B-chat.
- Removing the orthogonally-projected top-6 safety ranks (keeping top-96 utility ranks) yields ASR of 0.71 (vanilla), 0.97 (adv-suffix), 0.91 (adv-decoding) with 0.58 zero-shot accuracy.
- Removing the top-1 safety rank or top safety neurons drops Llama2-7B-chat accuracy to about 0.35-0.38, showing top safety regions also carry utility.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove safety-critical neurons isolated by SNIP set difference | Llama2-7B-chat · AdvBench eval, set difference (p=6, q=5) | ASR Adv-Suffix: 0.99 | Helped |
| remove safety-critical neurons isolated by SNIP set difference | Llama2-7B-chat · Six-task average zero-shot accuracy, set difference (p=6, q=5) | accuracy: 0.53 | Hurt |
| remove safety-critical ranks orthogonal to utility ranks | Llama2-7B-chat · AdvBench eval, 6 safety ranks removed and 96 utility ranks retained | ASR Vanilla: 0.71 | Helped |
| remove safety-critical ranks orthogonal to utility ranks | Llama2-7B-chat · Six-task average zero-shot accuracy, 6 safety ranks removed and 96 utility ranks retained | accuracy: 0.58 | No clear effect |
| remove the top safety-scored neurons instead of isolating safety from utility (ablation) | Llama2-7B-chat · Six-task average zero-shot accuracy | accuracy: 0.58 -> about $0.35$ | Hurt |
| remove the top safety-critical rank (ablation) | Llama2-7B-chat · Six-task average zero-shot accuracy | accuracy: $0.58$ -> $0.38$ | Hurt |
| prune neurons with the lowest safety importance scores rather than random neurons (ablation) | Llama2-7B-chat · AdvBench eval, adversarial decoding and suffix attacks | ASR | Helped |
| prune the least safety-relevant ActSVD ranks (ablation) | Llama2-7B-chat · AdvBench eval, adversarial decoding | ASR Adv-Decoding | Helped |
| use safety-short judgment-only data instead of safety-full responses for set-difference attribution (ablation) | Llama2-7B-chat · Set-difference pruning | ASR and average zero-shot accuracy | No clear effect |
| use safety-short instead of safety-full data when pruning the least safety-critical neurons (ablation) | Llama2-7B-chat · AdvBench eval, least-safety-critical neuron pruning | ASR Adv-Suffix and ASR Adv-Decoding | Mixed |
| freeze safety-critical neurons during Alpaca fine-tuning (n=10) (ablation) | Llama2-7B-chat · AdvBench eval, ASR Vanilla | ASR Vanilla: $0.53$ -> $0.23$ | Helped |
| freeze safety-critical neurons during Alpaca fine-tuning (n=50 or n=100) (ablation) | Llama2-7B-chat · AdvBench eval, ASR Vanilla | ASR Vanilla: $0.91$ -> $0.85$; $0.94$ -> $0.89$ | No clear effect |
Surprising findings
- Pruning neurons with the lowest safety importance scores slightly improves robustness to adversarial decoding attacks.
- Attention head probing accuracy above 0.95 does not translate into isolating safety-critical components; set difference outperforms it.
Limitations
- Only Llama2-chat models were studied due to limited availability of strong open safety-aligned models.
- Adversarial suffixes optimized on 7B models may not transfer to other models or modified models, adding variance to ASR_Adv-Suffix.
- Freezing safety-critical neurons during fine-tuning only helps with very few examples and high freeze fractions.
Experimental setup
Datasets: AdvBench Alpaca-Cleaned
Benchmarks: AdvBench BoolQ RTE HellaSwag WinoGrande ARC Challenge OpenBookQA
Metrics: attack success rate (ASR_Vanilla) ASR_Adv-Suffix ASR_Adv-Decoding average zero-shot accuracy
Baselines: Wanda (top) SNIP (top) attention head probing random pruning
Builds on: Wanda SNIP ActSVD GCG
Closest papers by meaning
- Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.92)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.91)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.91)
- The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions (ICML 2025 · similarity 0.90)
- A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models (NeurIPS 2026 · similarity 0.90)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.