Research map / Large language models / Language, safety and interpretability

The Sharp Directions Are Against You: Curvature Analysis of Activation Steering

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Task vectors and steering

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the steering Hessian and "sharp removal," which subtracts the top-curvature component of a CAA steering vector to make it robust, improving generation across tasks and models.

Problem

Contrastive activation addition (CAA) steering vectors are fragile: on many inputs they shift behavior in the wrong direction, and effectiveness is sensitive to layer and coefficient.

Approach

Define the steering Hessian (second derivative of behavioral loss w.r.t. the steering vector), decompose it into sharp (top-eigenvalue) and flat directions, and remove the sharp component from the CAA vector; the sharp part carries only 1-5% of squared norm but drives most of the steered-vs-unsteered distributional gap during generation.

Key results

• Steering along top Hessian eigenvectors reverses intended behavior in all 36 token-level conditions, with anti-steer rates of 53-100%.
• Sharp removal outperforms CAA in all 25 generation conditions; on Qwen refusal it raises refusal from 78% (CAA 36-44%) to 88-98%.
• Sharp directions account for only 1-5% of the CAA vector's squared norm while removing them keeps cosine above 0.974 and reduces directional curvature by a median 4x.

Surprising findings

• Steering along the top Hessian eigenvectors (sharp directions) reverses safety-aligned behavior on 100% of inputs for Llama refusal without adversarial optimization.
• Negating CAA is not sufficient on inverted conditions; the cleaned vector outperforms -v_CAA by 23-38 percentage points on refusal.

Limitations

• Mistral refusal keyword heuristic undercounts refusals (12% vs 52% judge baseline), so cross-model refusal comparisons are calibration-sensitive.
• Over-removal of sharp directions (large k) degrades performance, as seen on Llama sycophancy at k=25.
• Bounds hold only at alpha=1.0; generation-level improvements at higher alpha are empirical and not formally bounded.

Experimental setup

Datasets: RealToxicityPrompts AdvBench XSTest WikiText-2 contrastive CAA pairs (20 per task)

Benchmarks: AdvBench RealToxicityPrompts XSTest

Metrics: anti-steer rate probability shift condition number refusal rate honesty score sentiment score KL divergence false refusal rate

Baselines: CAA negated CAA random projection direction refusal direction (Arditi et al.)

Builds on: contrastive activation addition activation addition sharpness-aware minimization refusal direction

Closest papers by meaning

• Angular Steering: Behavior Control via Rotation in Activation Space (NeurIPS 2025 · similarity 0.90)
• Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior (NeurIPS 2026 · similarity 0.90)
• When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search (NeurIPS 2026 · similarity 0.90)
• Steer2Edit: From Activation Steering to Component-Level Editing (NeurIPS 2026 · similarity 0.89)
• Improving Instruction-Following in Language Models through Activation Steering (ICLR 2025 · similarity 0.88)
• Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models (NeurIPS 2026 · similarity 0.87)