Research map / Architectures and efficiency / Pruning and efficient tuning
Weight Updates as Activation Shifts: A Principled Framework for Steering
Dyah Adila, John Cooper, Alexander Yun, Avi Trost, Frederic Sala
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add orthogonality constraint to joint LoRA+adapter training (ablation) | joint weight+activation adaptation · GSM8K (Gemma-3-1b) | accuracy: 22.8 -> 24.4 | Improved |
| add orthogonality constraint to joint training (ablation) | joint adaptation · GSM8K (Qwen 3 4B) | accuracy: 37.9 -> 38.8 | Improved |
| add orthogonality constraint to joint training (ablation) | joint adaptation · Winogrande (Llama-3.2-1B) | accuracy: 56.9 -> 59.3 | Improved |
| add orthogonality constraint to joint training (ablation) | joint adaptation · Winogrande (Gemma-3-1b) | accuracy: 51.3 -> 50.9 | No clear effect |
| add orthogonality constraint to joint training (ablation) | joint adaptation · GSM8K (Llama-3.2-1B) | accuracy: 31.1 -> 31.1 | No clear effect |
| use post-block steering instead of ReFT at same 0.04% budget | post-block steering · ListOps (Llama-3.2-1B) | accuracy: 49.3 -> 65.4 | Improved |
| use post-block steering instead of ReFT | steering adapter · Winogrande (Qwen 3 4B) | accuracy: 64.9 -> 80.6 | Improved |
| use post-block steering instead of LoRA with 11x fewer parameters | steering adapter · Six-task average (Llama-3.2-1B) | average accuracy difference vs SFT: -2.2 -> -0.2 | Improved |
| use nonlinear instead of linear post-block adapter (r=16) (ablation) | post-block steering · AlpacaEval 2.0 (Llama-3.1-8B) | LC win rate: 11.00 -> 11.34 | Improved |
| use nonlinear instead of linear adapter (ablation) | post-block steering · GSM8K RL (R1-Distill-Qwen-1.5B) | Pass@1: 84.3 -> 84.7 | No clear effect |
| use post-block steering instead of LoRA in GRPO RL | post-block steering · GSM8K (R1-Distill-Qwen-1.5B) | Pass@1: 81.5 -> 84.7 | Improved |
| use post-block steering instead of LoRA r=8 | post-block steering · AlpacaEval 2.0 | LC win rate: 10.52 -> 11.00 | Improved |
Closest papers by meaning
- The Sharp Directions Are Against You: Curvature Analysis of Activation Steering (NeurIPS 2026 · similarity 0.90)
- When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search (NeurIPS 2026 · similarity 0.88)
- Angular Steering: Behavior Control via Rotation in Activation Space (NeurIPS 2025 · similarity 0.88)
- Steer2Edit: From Activation Steering to Component-Level Editing (NeurIPS 2026 · similarity 0.88)
- Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior (NeurIPS 2026 · similarity 0.87)
- Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
