Research map / Large language models / Language, safety and interpretability

On the Role of Attention Heads in Large Language Model Safety

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies a small set of safety-critical attention heads in aligned LLMs via a new head attribution metric (Ships) and two ablation methods (Undifferentiated Attention and Scaling Contribution), showing ablating them compromises refusal.

Problem

Existing LLM safety interpretability work focuses on neurons, representations, or residual streams and overlooks the role of multi-head attention in safety; prior head-ablation methods are judged insufficient for identifying safety heads.

Approach

Ablate attention heads by scaling their Query/Key matrices by a tiny epsilon, which makes softmax attention uniform (undifferentiated), or scaling the Value matrix to reduce contribution, and attribute safety to heads whose ablation most increases attack success rate, using a metric called Ships computed on harmful queries or datasets.

Key results

• Ablating the top Ships-identified heads in Vicuna-7b-v1.5 raises ASR by about +0.11 (MaliciousInstruct) and +0.33 (JailbreakBench) on average with Undifferentiated Attention (dataset-level attribution, 1-5 heads).
• On Llama-2-7b-chat, undifferentiated-attention ablation of safety heads raises ASR above 0.3 even with the official system template, from 0 without ablation.
• Ablating safety heads also weakens course-correction: under affirmative 'Sure, here is' prefilling, Llama-2-7b-chat ASR on AdvBench rises from 0.35 to 0.68 with Undifferentiated Attention.

Measured effects reported in the paper

Surprising findings

• Safety of aligned models appears to depend heavily on in-context learning from system prompts, so the paper argues system prompts should be excluded when attributing inherent safety parameters.
• Safety-critical heads show a long-tailed distribution, suggesting only a small number of heads carry most safety function.

Limitations

• Rule-based keyword ASR judgment, with keyword list expanded by the authors and filtering heuristics that may misclassify outputs.
• Analysis limited to a few 7-8B models (Llama-2-7b-chat, Vicuna-7b-v1.5, Llama-3-8B).
• Relation between attention heads and safety capabilities beyond direct refusal (e.g., course-correction) is only preliminarily explored.

Experimental setup

Datasets: AdvBench JailbreakBench MaliciousInstruct

Benchmarks: AdvBench JailbreakBench MaliciousInstruct

Metrics: attack success rate Ships

Baselines: Safety-rank

Builds on: Safety-rank URIAL

Closest papers by meaning

• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.92)
• Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.91)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.90)
• SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.89)
• Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.89)
• Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements (ICLR 2025 · similarity 0.89)