Research map / Large language models / Language, safety and interpretability

A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that pinning or anchoring a single MLP neuron (a "refusal neuron") bypasses safety alignment on JailbreakBench across seven Qwen3 and Llama-3.1 models, and that amplifying a single "suicide neuron" injects suicide content into benign prompts.

Problem

Whether safety alignment is distributed across many weights or can be bypassed, and whether harmful knowledge is localized, at the granularity of a single MLP neuron rather than directions or neuron sets.

Approach

A gradient-activation score over MLP neurons, followed by empirical reranking on a validation set, selects a single neuron whose pinned or anchored activation suppresses refusal at one layer; the same single-neuron amplification idea is applied to a concept (suicide) neuron.

Key results

• Single refusal neuron with constant or anchor intervention averages 91.7% (constant) and 90.4% (anchor) ASR under the LLM judge on JailbreakBench across seven models, versus 93.1% for Arditi et al.
• Anchor variant limits MMLU degradation to -0.6% on average versus -8.8% for the constant intervention.
• Llama-3.1-8B refusal neuron achieves AUROC 0.969 on XSTest, with recall 0.950 versus 0.834 for Llama-Guard-3-8B, at lower precision (0.848 vs 0.949).

Surprising findings

• Best residual-stream single feature reaches only 45% ASR on Llama-3.1-8B and 39% on Qwen3-8B, far below MLP neurons.
• Refusal neuron signal is already discriminative in base (pre-alignment) Qwen3 checkpoints, suggesting alignment rewires when neurons fire rather than creating them.

Limitations

• Concept neurons demonstrated only for suicide-related content on three Qwen3 models.
• Constant intervention substantially degrades MMLU (-8.8% average); anchor variant needed to preserve capability.
• Optimal multiplier selected by empirical sweep on validation set; no principled selection method.

Experimental setup

Datasets: AdvBench MaliciousInstruct TDC2023 Alpaca HarmBench JailbreakBench XSTest The Pile (uncopyrighted subset) MMLU GSM8K

Benchmarks: JailbreakBench HarmBench XSTest MMLU GSM8K

Metrics: attack success rate AUROC accuracy F1 precision recall

Baselines: Arditi refusal direction ablation NeuroStrike residual-stream single-feature attack Llama-Guard-3-8B

Builds on: refusal direction ablation (Arditi et al.) NeuroStrike SAE feature analysis

Closest papers by meaning

• Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.92)
• Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications (ICML 2024 · similarity 0.90)
• Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.88)
• The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions (ICML 2025 · similarity 0.88)
• AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.88)
• Redefining Experts: Interpretable Decomposition of Language Models for Toxicity Mitigation (NeurIPS 2025 · similarity 0.88)