Research map / Large language models / Alignment and preferences
Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiPO, which optimizes a single layer steering vector directly against contrastive free-form preference pairs using a DPO-style bi-directional objective, yielding stronger and more controllable persona, truthfulness, hallucination, and jailbreak steering.
Problem
Steering vectors extracted from activation differences on contrastive multiple-choice prompts often fail to change the model's actual generations, especially for alignment-related behaviors such as jailbreaking, truthfulness, and hallucination.
Approach
Instead of averaging activation differences of prompts, optimize the steering vector itself with a DPO-style loss so it directly raises the generation likelihood of target-behavior responses over opposite-behavior responses; a random sign coefficient d lets the same procedure yield both the behavior and its opposite, so scaling by +/- controls intensity and direction.
Key results
- On Llama-2-7b-chat-hf, BiPO steering with a jailbreak vector reaches 73% ASR on malicious questions versus 0% for CAA and freeform baselines.
- Under GCG adversarial suffixes, subtracting the BiPO vector reduces ASR from 16% to 0%.
- Steering vectors keep MMLU accuracy roughly unchanged at multipliers -1 and +1 across the four AI persona behaviors on Llama-2-7b-chat-hf and Mistral-7B-Instruct-v0.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the BiPO steering vector | Llama-2-7B-chat-hf · AdvBench malicious questions | ASR: 0% -> 73% | Helped |
| Subtract the BiPO steering vector | Llama-2-7B-chat-hf · AdvBench malicious questions with GCG adversarial suffixes | ASR: 16% -> 0% | Helped |
| Use CAA or Freeform steering vectors | Llama-2-7B-chat-hf · AdvBench malicious questions | ASR: 0% -> 0% | No clear effect |
| Use the BiPO steering vector to steer TruthfulQA truthfulness | Llama-2-7B-chat-hf · TruthfulQA | accuracy (MC1 and MC2) | Helped |
| Use BiPO rather than baseline steering vectors | Llama-2-7B-chat-hf · AI persona open-ended generation | GPT-4 behavioral score | Helped |
| Vary the direction and magnitude of the BiPO vector | Llama-2-7B-chat-hf · AI persona control | GPT-4 behavioral score | Helped |
| Apply AI-persona steering vectors at multipliers -1 or 1 | Mistral-7B-Instruct-v0.2 · MMLU | accuracy | No clear effect |
| Combine Power-seeking and Wealth-seeking vectors | Llama-2-7B-chat-hf · Power-seeking and Wealth-seeking | behavioral score: 2.38 -> 2.67; 1.89 -> 2.30 | Helped |
| Combine Corrigible-less-HHH and Hallucination vectors | Llama-2-7B-chat-hf · Corrigible-less-HHH and Hallucination | behavioral score: 1.62 -> 1.94; 2.64 -> 3.24 | Helped |
| Optimize the BiPO steering vector at the 15th layer rather than other tested layers (ablation) | Llama-2-7B-chat-hf · AI persona steering | steering effect | Helped |
| Increase the number of BiPO training epochs (ablation) | Llama-2-7B-chat-hf · AI persona steering | steering effect | Mixed |
Surprising findings
- CAA vectors often fail to steer the model's completion even when the appended answer is applied, with inconsistency across nearly all jailbreak training samples.
Limitations
- Steering applied to only a single transformer layer; multi-layer steering is left to future work.
- Evaluation relies on GPT-4 judging of open-ended generations.
- Hyperparameters such as layer and epochs vary by dataset and model.
Experimental setup
Datasets: Anthropic Model-Written Evaluations (Advanced AI Risk) TruthfulQA Unprompted hallucination dataset AdvBench
Benchmarks: Anthropic Model-Written Evaluations (power-seeking, wealth-seeking, corrigible-less-HHH, survival-instinct) TruthfulQA unprompted hallucination AdvBench MMLU
Metrics: GPT-4 behavioral score (1-4) MC1 MC2 attack success rate MMLU accuracy
Baselines: CAA Freeform steering vector
Builds on: Contrastive Activation Addition Direct Preference Optimization Activation Addition Inference-Time Intervention
Closest papers by meaning
- Improved Representation Steering for Language Models (NeurIPS 2025 · similarity 0.92)
- Steering When Necessary: Flexible Steering Large Language Models with Backtracking (NeurIPS 2025 · similarity 0.90)
- Analysing the Generalisation and Reliability of Steering Vectors (NeurIPS 2024 · similarity 0.89)
- When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search (NeurIPS 2026 · similarity 0.89)
- Focus On This, Not That! Steering LLMs with Adaptive Feature Specification (ICML 2025 · similarity 0.88)
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.