Research map / Reinforcement learning / RL from feedback
Steering Vectors as a Training Signal in LLM Post-Training
Tiejin Chen, Maunil R Vyas, Huaiyuan Yao, Hua Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Investigates whether steering vectors, normally used as inference-time controls, can also guide parameter updates during LLM post-training across SFT, distillation, and RLVR.
Problem
Steering vectors are usually applied post hoc to shift LLM behavior at inference time; it is unclear whether these residual-stream directions can serve as a training signal during post-training.
Approach
Uses steering vectors as a signal that guides parameter updates during post-training, rather than only as post-hoc controls, studied across supervised learning (SFT and distillation) and reinforcement learning with verifiable rewards.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add steering-vector injection (per-sample Slerp) during teacher-forced SFT | LoRA SFT on Qwen models · GSM8K, MATH, MBPP, StrategyQA | accuracy (%): Qwen2.5-1.5B GSM8K 59.14 -> 64.14 | Improved |
| add SV injection during SFT with default mid-layer on a different family | Llama-3.2-3B SFT · GSM8K, MATH, MBPP, StrategyQA | accuracy (%): mean gain +0.23pp for Per-sample Slerp | No clear effect |
| add reference-direction SV during SFT on code | Qwen2.5-1.5B SFT · MBPP | accuracy (%): 47.00 -> 45.60 | Hurt |
| inject Gaussian noise instead of the reference-minus-prediction SV (ablation) | SV-SFT · GSM8K | accuracy (%): 59.14 -> 54.82 | Hurt |
| use per-sample direction with Slerp instead of per-sample additive injection (ablation) | SV-SFT · GSM8K | accuracy (%): 62.93 -> 64.14 | Improved |
| inject at three layers with uniform alpha 0.50 instead of a single layer (ablation) | SV-SFT · GSM8K | accuracy (%): 61.11 -> 60.85 | Hurt |
| increase Slerp alpha to 1.0 (ablation) | Per-sample Slerp SFT · GSM8K | accuracy (%): collapses to 0% | Hurt |
| keep SV at inference time on an SV-trained or plain checkpoint (ablation) | Plain-SFT checkpoint · GSM8K | accuracy (%): 59.14 -> 44.43 | Hurt |
| restrict SV injection to the first K answer tokens or gate by sample or confidence (ablation) | SV-SFT · GSM8K | accuracy (%): Slerp first-K=64 58.61 vs full answer 64.14 | Hurt |
| distill from thinking-mode teachers with SV injection | Qwen2.5-1.5B student · MATH distillation | accuracy (%): 21.20 -> 33.60 (Qwen3-8B thinking) | Improved |
| add SV injection to distillation from non-thinking teachers | Qwen distillation students · GSM8K, MATH distillation | accuracy (%): Qwen2.5-1.5B MATH 41.60 -> 42.20 | No clear effect |
| add SV-Rescue advantage repair on wrong rollouts to GRPO | GRPO with Qwen/Llama students · GSM8K and DeepScaleR validation | accuracy (%): Llama-3.2-3B GSM8K 81.83 -> 84.49 | Improved |
Closest papers by meaning
- Training-Free Task Vectors for LLM Behavioral Control (NeurIPS 2026 · similarity 0.88)
- Learning to Steer: Input-dependent Steering for Multimodal LLMs (NeurIPS 2025 · similarity 0.87)
- The Best Instruction-Tuning Data are Those That Fit (NeurIPS 2025 · similarity 0.86)
- The Sharp Directions Are Against You: Curvature Analysis of Activation Steering (NeurIPS 2026 · similarity 0.85)
- Inverted Detection and Control in Steering Vectors (NeurIPS 2026 · similarity 0.85)
- Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
