Research map / Large language models / Instruction and fine-tuning

Refusal in Language Models Is Mediated by a Single Direction

NeurIPS 2024 · Large language models › Instruction and fine-tuning · approach: Gradient-free adaptation and merging · 57 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows refusal in 13 open-source chat models (1.8B–72B) is mediated by a single difference-in-means direction, and uses directional ablation folded into weights (weight orthogonalization) as a cheap white-box jailbreak, plus a mechanistic look at adversarial suffixes.

Problem

Refusal behavior in safety fine-tuned chat LLMs is widely observed but its internal mechanism is poorly understood, and its robustness to white-box manipulation is unclear.

Approach

Refusal is controlled by one residual-stream direction, extracted as the difference-in-means between harmful and harmless activations; ablating it disables refusal and adding it induces refusal, and orthogonalizing weight matrices against it yields a rank-one weight-edit jailbreak that requires no gradient optimization.

Key results

• Directional ablation of a single refusal direction bypasses refusal across 13 chat models up to 72B, while activation addition induces refusal on harmless Alpaca instructions.
• Weight orthogonalization achieves HarmBench ASR of 22.6% (Llama-2 7B with system prompt, 79.9% without) and 79.2% to 84.3% for Qwen 7B and 14B, comparable to prompt-specific GCG on Qwen models.
• Orthogonalized models stay largely within confidence intervals of baselines on MMLU, ARC, and GSM8K; TruthfulQA accuracy consistently drops.

Measured effects reported in the paper

Surprising findings

• Llama-2 jailbreak ASR is highly sensitive to the system prompt (22.6% with vs 79.9% without), while Qwen ASR is stable across system prompts.
• Adversarial suffixes suppress the refusal direction's expression and shift head attention from the harmful instruction to the suffix, contrary to prior claims that representations are unaltered by suffixes.

Limitations

• Findings may not generalize to proprietary or larger future models; only open-source chat models tested.
• Direction extraction relies on heuristics and is an existence proof rather than an optimal method.
• Mechanistic analysis of adversarial suffixes covers a single model (Qwen 1.8B Chat) and a single suffix.

Experimental setup

Datasets: AdvBench MaliciousInstruct TDC2023 HarmBench Alpaca JailbreakBench

Benchmarks: JailbreakBench HarmBench MMLU ARC GSM8K TruthfulQA WinoGrande TinyHellaSwag

Metrics: refusal score (substring match) safety score (Llama Guard 2) attack success rate refusal log-odds metric KL divergence CE loss accuracy

Baselines: GCG GCG-M GCG-T AutoPrompt PAIR Human DAN templates Direct response

Builds on: difference-in-means feature directions GCG HarmBench Llama Guard 2

Closest papers by meaning

• Latent-space Attacks for Refusal Evasion in Language Models (NeurIPS 2026 · similarity 0.92)
• The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence (ICML 2025 · similarity 0.91)
• Refusal Direction is Universal Across Safety-Aligned Languages (NeurIPS 2025 · similarity 0.90)
• Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation (ICLR 2025 · similarity 0.90)
• Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuning (ICML 2025 · similarity 0.90)
• Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.88)