Research map / Large language models / Language, safety and interpretability
Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies sparse "safety neurons" in LLMs via inference-time activation contrasting (DPO vs SFT) and dynamic activation patching, showing ~5% of neurons can restore over 90% of safety and that safety and helpfulness neurons overlap, explaining alignment tax, and proposes a neuron-based pre-generation safeguard.
Problem
Understanding the internal mechanisms of safety alignment in LLMs at the neuron level, and explaining the alignment tax, given that existing mechanistic interpretability methods target few-token tasks and attention heads rather than open-ended safety behavior.
Approach
Locate candidate neurons by comparing RMS activation differences between SFT and DPO models on the same generated text (inference-time activation contrasting), then verify causal effect on open-ended generation by dynamically patching the aligned model's neuron activations into the unaligned model at every decoding step (dynamic activation patching); enhancement rather than ablation is used to fin
Key results
- Patching about 5% of neurons from DPO into SFT recovers over 90% of DPO's safety performance across benchmarks, sometimes exceeding full DPO (e.g., Llama2 SFT Beavertails cost -11.9 vs DPO -11.8).
- Patching top safety neurons yields causal effects of roughly +63 to +134% on red-teaming benchmarks, versus near-zero or inconsistent effects for Pruning and SN-Tune baselines, with general benchmark changes around -0.04 or less.
- A logistic regression classifier on 1,500 SFT activations at the prompt's last token predicts harmful outputs with 76.2% average accuracy, enabling a pre-generation safeguard.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Patch SFT with DPO activations on top safety neurons | Llama2-7B SFT · Beavertails | causal effect (%): +101 | Helped |
| Patch SFT with DPO activations on top safety neurons | Mistral-7B SFT · RedTeam | causal effect (%): +80 | Helped |
| Patch SFT with DPO activations on top safety neurons | Gemma-7B SFT · HarmBench | causal effect (%): +79 | Helped |
| Patch SFT with DPO activations on top safety neurons | Qwen2.5-3B SFT · JailBreakLLMs | causal effect (%): +83 | Helped |
| Use top safety neurons rather than random neurons for DPO-activation patching | Base and SFT models · Beavertails | cost score | Helped |
| Use top 5% safety neurons rather than random neurons for DPO-activation patching | Base and SFT models · Beavertails | cost score: 1.15×10⁻⁶ to 1.67×10⁻¹⁸ | Helped |
| Patch with safety-neuron activations | Base and SFT models · general benchmarks | performance change: -0.04 to +0.05 | No clear effect |
| Use helpfulness-DPO activations instead of safety-DPO activations on shared neurons | Llama2 safety DPO · Beavertails | safety score change: 7.3 | Hurt |
| Use safety-DPO activations instead of helpfulness-DPO activations on shared neurons | Llama2 helpfulness DPO · Beavertails | safety score change: 10.1 | Helped |
| Use helpfulness-DPO activations instead of safety-DPO activations on shared neurons | Mistral safety DPO · Beavertails | safety score change: 6.6 | Hurt |
| Use safety-DPO activations instead of helpfulness-DPO activations on shared neurons | Mistral helpfulness DPO · Beavertails | safety score change: 10.7 | Helped |
| Identify neurons by comparing SFT and DPO rather than other model pairs (ablation) | Safety-neuron identification · Beavertails | cost score | Helped |
Surprising findings
- Ablation-based neuron identification methods (Pruning, SN-Tune) are not causally effective under the enhancement-based verification, sometimes degrading or inconsistently affecting safety.
- Top safety neurons' value vectors project to tokens without explicit safety content (e.g., food, conjunctions, closing brackets), unlike previously reported toxic vectors.
Limitations
- Causal effect is measured mainly with a single cost model (beaver-7b-v1.0-cost) as safety metric, with GPT-4 correlation only on one model; alignment tax analysis uses a limited set of preference datasets.
- Safety neurons are shown to exist in four models of 3-7B scale; generality to larger models is untested.
- Mechanism of the identified neurons remains unclear; value-vector token projections show no obvious safety semantics.
Experimental setup
Datasets: ShareGPT HH-RLHF RewardBench Stanford Human Preferences (SHP) H4 Stack Exchange Preferences IEFeedback Beavertails RedTeam HarmBench JailBreakLLMs Wikitext-2 MMLU
Benchmarks: Beavertails RedTeam HarmBench JailBreakLLMs Wikitext-2 MMLU GSM8K BBH TruthfulQA
Metrics: cost score (beaver-7b-v1.0-cost) causal effect accuracy perplexity Spearman correlation GPT-4 score
Baselines: Pruning (Wanda-based parameter pruning) SN-Tune Random neurons
Builds on: DPO activation patching Invertible Adapters with Activation Alignment (IA3-style PEFT) Wanda Circuit breakers
Closest papers by meaning
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.93)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.93)
- Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements (ICLR 2025 · similarity 0.92)
- Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications (ICML 2024 · similarity 0.92)
- A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models (NeurIPS 2026 · similarity 0.92)
- Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.