Research map / Large language models / Language, safety and interpretability
Analysing the Generalisation and Reliability of Steering Vectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically evaluates the in-distribution reliability and out-of-distribution generalisation of contrastive activation addition steering vectors across 40 Model-Written Evals datasets and two chat models, finding high per-sample variance, anti-steerability driven by answer-position/token biases, and imperfect OOD transfer.
Problem
Steering vectors are promising for inference-time behaviour control, but their in-distribution reliability across inputs and out-of-distribution generalisation across system and user prompt changes are unknown.
Approach
Measures per-sample steerability and its variance rather than only aggregate effects, identifies a steerability bias (position/token preference in the steering effect), and evaluates OOD generalisation via prompt injections in system and user messages, using a relative steerability metric and propensity similarity analysis.
Key results
- Steerability varies widely per sample across all evaluated behaviours, with some datasets having almost 50% anti-steerable inputs.
- Steerability is strongly correlated between Llama-2-7b and Qwen-1.5-14b in-distribution (Spearman rho = 0.769), suggesting it is mostly a dataset property.
- OOD steerability is reasonably correlated with ID steerability (rho = 0.891 for Llama-2-7b, 0.694 for Qwen-1.5-14b), but generalisation is imperfect and generally worse OOD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transfer steering vectors from in-distribution prompts to shifted prompts | CAA steering vectors · Prompt distribution shifts | Steerability | Hurt |
| Use prompt settings with more similar unsteered model propensity | CAA steering vectors · Prompt distribution shifts | Relative steerability: -0.26; -0.46 | Helped |
| Extract steering vectors from datasets with similar behavior across models | CAA steering vectors · In-distribution and out-of-distribution datasets | Steerability: 0.769; 0.586 | No clear effect |
| Change the positive option's token or position (A/B or Yes/No) | CAA steering vectors · In-distribution MWE datasets | Per-sample steerability | Mixed |
| Apply steering to individual examples rather than relying on aggregate behavior | CAA steering vectors · In-distribution MWE datasets | Per-sample steerability | Mixed |
| Vary the multiplier range used to calculate steerability (ablation) | CAA steering vectors · In-distribution steerability | Steerability | No clear effect |
| Use out-of-distribution rather than in-distribution prompts | CAA steering vectors · Steerability variance | Variance in steerability: 0.535; 0.341 | No clear effect |
| Use a steering vector extracted from another dataset variation | CAA steering vectors · Out-of-distribution prompt variations | Steering-vector cosine similarity: -0.63; -0.86 | Helped |
Surprising findings
- Spurious steerability biases (answer position A/B or token Yes/No) are present despite balanced training data and explain a large share of per-sample steerability variance.
- Steering vectors can produce the opposite of the intended behaviour for a large fraction of inputs on many datasets.
Limitations
- Only multiple-choice format evaluated, not open-ended generation.
- Only two main models analysed (Llama-2-7b-chat and Qwen-1.5-14b-chat) in the main analysis.
- Unclear whether low steerability reflects dataset quality, non-linear representation, or multiple sub-behaviours.
Experimental setup
Datasets: Model-Written Evaluations (MWE) persona and AI risk datasets TruthfulQA sycophancy dataset
Benchmarks: Model-Written Evaluations (40 datasets) TruthfulQA sycophancy dataset
Metrics: logit-difference propensity steerability (slope of mean-squares line fit) relative steerability Spearman rank correlation fraction of anti-steerable examples
Baselines: Contrastive Activation Addition
Builds on: Contrastive Activation Addition Model-Written Evals representation engineering linear representation hypothesis
Closest papers by meaning
- To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Models (ICML 2025 · similarity 0.89)
- Model Whisper: Steering Vectors Unlock Large Language Models’ Potential in Test-Time (AAAI 2026 · similarity 0.89)
- Steering When Necessary: Flexible Steering Large Language Models with Backtracking (NeurIPS 2025 · similarity 0.89)
- Improved Representation Steering for Language Models (NeurIPS 2025 · similarity 0.89)
- Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization (NeurIPS 2024 · similarity 0.89)
- Improving Instruction-Following in Language Models through Activation Steering (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.