AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Steering Vectors as a Training Signal in LLM Post-Training

Tiejin Chen, Maunil R Vyas, Huaiyuan Yao, Hua Wei

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Investigates whether steering vectors, normally used as inference-time controls, can also guide parameter updates during LLM post-training across SFT, distillation, and RLVR.

Problem

Steering vectors are usually applied post hoc to shift LLM behavior at inference time; it is unclear whether these residual-stream directions can serve as a training signal during post-training.

Approach

Uses steering vectors as a signal that guides parameter updates during post-training, rather than only as post-hoc controls, studied across supervised learning (SFT and distillation) and reinforcement learning with verifiable rewards.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add steering-vector injection (per-sample Slerp) during teacher-forced SFTLoRA SFT on Qwen models · GSM8K, MATH, MBPP, StrategyQAaccuracy (%): Qwen2.5-1.5B GSM8K 59.14 -> 64.14Improved
add SV injection during SFT with default mid-layer on a different familyLlama-3.2-3B SFT · GSM8K, MATH, MBPP, StrategyQAaccuracy (%): mean gain +0.23pp for Per-sample SlerpNo clear effect
add reference-direction SV during SFT on codeQwen2.5-1.5B SFT · MBPPaccuracy (%): 47.00 -> 45.60Hurt
inject Gaussian noise instead of the reference-minus-prediction SV (ablation)SV-SFT · GSM8Kaccuracy (%): 59.14 -> 54.82Hurt
use per-sample direction with Slerp instead of per-sample additive injection (ablation)SV-SFT · GSM8Kaccuracy (%): 62.93 -> 64.14Improved
inject at three layers with uniform alpha 0.50 instead of a single layer (ablation)SV-SFT · GSM8Kaccuracy (%): 61.11 -> 60.85Hurt
increase Slerp alpha to 1.0 (ablation)Per-sample Slerp SFT · GSM8Kaccuracy (%): collapses to 0%Hurt
keep SV at inference time on an SV-trained or plain checkpoint (ablation)Plain-SFT checkpoint · GSM8Kaccuracy (%): 59.14 -> 44.43Hurt
restrict SV injection to the first K answer tokens or gate by sample or confidence (ablation)SV-SFT · GSM8Kaccuracy (%): Slerp first-K=64 58.61 vs full answer 64.14Hurt
distill from thinking-mode teachers with SV injectionQwen2.5-1.5B student · MATH distillationaccuracy (%): 21.20 -> 33.60 (Qwen3-8B thinking)Improved
add SV injection to distillation from non-thinking teachersQwen distillation students · GSM8K, MATH distillationaccuracy (%): Qwen2.5-1.5B MATH 41.60 -> 42.20No clear effect
add SV-Rescue advantage repair on wrong rollouts to GRPOGRPO with Qwen/Llama students · GSM8K and DeepScaleR validationaccuracy (%): Llama-3.2-3B GSM8K 81.83 -> 84.49Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy