Research map / Large language models / Language, safety and interpretability

LLMs Encode Harmfulness and Refusal Separately

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows LLMs encode harmfulness at the instruction-final token and refusal at the post-instruction token, using a harmfulness direction, a reply-inversion causal test, jailbreak analysis, and a training-free "Latent Guard" classifier competitive with Llama Guard 3 8B.

Problem

Whether LLMs internally represent harmfulness as a concept distinct from refusal behavior, since prior work assumed the refusal direction encodes harmfulness.

Approach

Hidden states at the last instruction token cluster by harmfulness while those at the last post-instruction token cluster by refusal; a harmfulness direction extracted at the instruction token, unlike the refusal direction, flips judgments in a reply-inversion task, and the resulting latent belief score is used as a training-free guard.

Key results

• Steering harmless instructions along the harmfulness direction at layer 9 of Llama3 yields a 94% refusal rate, while the refusal direction reaches 100% at layer 11.
• Latent Guard on Qwen2 reaches 75% accuracy on persuasion jailbreaks vs 17.8% for Llama Guard 3 8B.
• Latent Guard is worse than Llama Guard 3 on ToxicChat and OpenAI Moderation without in-domain examples, but surpasses it on most taxonomies after adding 50 in-domain examples.

Measured effects reported in the paper

Surprising findings

• Removing post-instruction tokens drops Llama2 refusal rate on AdvBench from 100% to 85.3%, and Llama3 from 96% to 58.9%.
• Refused harmless instructions have negative harmfulness belief scores, meaning the model internally deems them harmless despite refusing.

Limitations

• Only 7B and 8B open-source models were tested, so generalization to larger models is unknown.
• Latent Guard underperforms Llama Guard 3 on out-of-domain datasets such as ToxicChat and OpenAI Moderation without in-domain examples.
• The study is an existence proof and does not identify which layers or neurons implement harmfulness and refusal.

Experimental setup

Datasets: AdvBench JBB SORRY-Bench Alpaca XSTest CATQA

Benchmarks: AdvBench JBB SORRY-Bench XSTest ToxicChat OpenAI Moderation Evaluation Dataset

Metrics: refusal rate classification accuracy cosine similarity

Baselines: Llama Guard 3 8B

Builds on: refusal direction difference-in-means

Closest papers by meaning

• The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions (ICML 2025 · similarity 0.90)
• The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence (ICML 2025 · similarity 0.90)
• Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs (NeurIPS 2026 · similarity 0.89)
• Latent-space Attacks for Refusal Evasion in Language Models (NeurIPS 2026 · similarity 0.89)
• WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · similarity 0.89)
• Refusal Direction is Universal Across Safety-Aligned Languages (NeurIPS 2025 · similarity 0.89)