Research map / Large language models / Language, safety and interpretability
Improved Representation Steering for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RePS, a reference-free bidirectional preference optimization objective (built on SimPO) for training intervention-based steering methods (steering vectors, LoRA, ReFT), evaluated on AxBench with Gemma-2 and Gemma-3 models from 2B to 27B for steering and suppression.
Problem
Intervention-based steering methods (steering vectors, LoRA, ReFT) underperform prompting, partly because their language-modeling training objectives ignore preference signals; prior preference-based steering (BiPO) is reference-anchored and struggles at scale.
Approach
A reference-free bidirectional preference objective: positive steering up-weights the steered response likelihood, while negative steering (via a nulling intervention or sampled negative factors) suppresses the concept; combined with factor sampling during training and SimPO-style length normalization.
Key results
- On Gemma-2-2B (D_L10), RePS-trained steering vectors score 0.756 vs 0.663 for Lang. and 0.199 for BiPO; prompting scores 0.698.
- On Gemma-3-27B (D_100), RePS LoReFT scores 0.436 vs 0.129 for Lang.; RePS SV reaches 1.269 vs prompt 1.547.
- Under instruction-following attacks, RePS SV suppression on Gemma-3-27B scores 0.734 vs 0.700 for Lang. and 0.135-0.275 for appended/prepended prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train steering interventions with RePS instead of the language-modeling objective | SV rank 1 on Gemma-2-2B · AxBench concept steering | steering score: 0.663 -> 0.756; 0.568 -> 0.606 | Helped |
| Train steering interventions with RePS instead of the language-modeling objective | SV rank 1 on Gemma-2-9B · AxBench concept steering | steering score: 0.788 -> 0.892; 0.580 -> 0.624 | Helped |
| Train steering interventions with RePS instead of the language-modeling objective | SV rank 1 on Gemma-3-12B and Gemma-3-27B · AxBench concept steering | steering score: 1.219 -> 1.230; 1.228 -> 1.269 | Helped |
| Use RePS instead of BiPO | intervention-based steering · AxBench concept steering | steering score: 0.199 -> 0.756 | Helped |
| Train LoRA rank 4 with RePS instead of the language-modeling objective | Gemma-2-2B · AxBench concept steering | steering score: 0.710 -> 0.798; 0.723 -> 0.793 | Helped |
| Train LoReFT rank 4 with RePS instead of the language-modeling objective | Gemma-3-27B · AxBench concept steering | steering score: 0.129 -> 0.436 | Helped |
| Train LoReFT rank 4 with RePS instead of the language-modeling objective | Gemma-3-12B · AxBench concept steering | steering score: 0.714 -> 0.651 | Hurt |
| Use RePS instead of the language-modeling objective for negative-factor suppression | SV rank 1 on Gemma-2 · AxBench concept suppression | suppression score: 1.211 -> 1.205; 0.936 -> 0.929; 1.154 -> 1.100; 0.862 -> 0.834 | Mixed |
| Use RePS instead of the language-modeling objective for negative-factor suppression | SV rank 1 on Gemma-3 · AxBench concept suppression | suppression score: 0.912 -> 1.035; 0.940 -> 1.031 | Helped |
| Use intervention-based RePS suppression rather than a prepended system prompt | SV rank 1 · 20 rule-based concepts under instruction-following attacks | suppression score: 0.870 vs 0.427; 0.734 vs 0.275 | Helped |
| Sample steering factors during training (ablation) | steering-vector training · Hyperparameter-tuning runs | variance of steering scores | Helped |
| Add a learned scalar projection gate for each embedding (ablation) | SV interventions · Offline steering evaluation | performance | No clear effect |
Surprising findings
- BiPO performs very poorly on Gemma-2 steering (around 0.07-0.22), far below the language modeling objective.
- Prompting remains the best approach for concept suppression overall, while intervention-based suppression is more robust to jailbreak attacks.
Limitations
- LoRA and LoReFT underperform rank-1 steering vectors on larger models, with LoReFT failing almost catastrophically on Gemma-3-27B suppression.
- Hyperparameter tuning used a small development set of three concepts and a reduced grid due to compute limits; ReFT was excluded for 27B.
- Relies on AxBench datasets and remote LM judges, which may not be optimal and may favor longer answers.
Experimental setup
Datasets: AxBench Concept500 AlpacaEval IFEval-derived rule-based concepts
Benchmarks: AxBench Concept500 AlpacaEval (suppression) IFEval-based rule concepts (suppression under attack)
Metrics: concept score instruct score fluency score harmonic-mean steering score suppression score
Baselines: Prompt Lang. (language modeling objective) BiPO LoReFT ReFT-r1 DiffMean SAE
Builds on: SimPO BiPO DPO AxBench LoReFT
Closest papers by meaning
- Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization (NeurIPS 2024 · similarity 0.92)
- Steering When Necessary: Flexible Steering Large Language Models with Backtracking (NeurIPS 2025 · similarity 0.90)
- Analysing the Generalisation and Reliability of Steering Vectors (NeurIPS 2024 · similarity 0.89)
- Representation Surgery: Theory and Practice of Affine Steering (ICML 2024 · similarity 0.89)
- Focus On This, Not That! Steering LLMs with Adaptive Feature Specification (ICML 2025 · similarity 0.88)
- Improving Instruction-Following in Language Models through Activation Steering (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.