Research map / Large language models / Instruction and fine-tuning
Activation Space Interventions Can Be Transferred Between Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that steering vectors can be transferred between LLMs (same and cross-family, different sizes) by training autoencoder mappings between their activation spaces, applied to backdoor removal, refusal, and a new corrupted-capabilities task.
Problem
Whether safety-relevant activation interventions (steering vectors for backdoor removal, refusal) learned on one LLM can be transferred to another LLM via mappings between activation spaces, without analyzing the target model's activations directly.
Approach
Train a dense autoencoder (ReLU encoder, linear decoder) mapping source-model layer activations into target-model layer activation space, so steering vectors or whole activations from a small model can be substituted into a larger model; also map base and fine-tuned activations to toggle behaviors.
Key results
- Mapped steering in Qwen 0.5B to 1.5B IHY task reaches LLM-judge 2.6 vs 0.0 for mean-ablated, with mapped completions winning 100% of comparisons on several datasets.
- Corrupted capabilities task: mapped vector achieves only 6.34% success, indicating limited transfer of knowledge-bearing behavior.
- Single-layer patching of fine-tuned Qwen 0.5B activations from base model reduces backdoor trigger rate by up to 60%; the autoencoder mapper has 0.32% of the parameters of Qwen 0.5B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use autoencoder-mapped activations from Qwen 0.5B instead of mean-ablated activations | Qwen 1.5B · IHY RLHF | LLM-Judge: 0.0 → 2.6 | Helped |
| Transfer activations from Qwen 0.5B rather than use Gemma 2B as the source | Llama 3B · IHY RLHF | LLM-Judge: 1.2 → 3.0 | Helped |
| Use Qwen as the source instead of Gemma, whose tokenizer differs from Llama | Llama 3B · IHY RLHF | COH: 2.40 → 3.70 | Helped |
| Use Qwen 1.5B rather than Qwen 0.5B as the source | Llama 3B · IHY RLHF | KL-Div: 6.97 → 5.63 | Helped |
| Use a nonlinear autoencoder instead of an affine map (ablation) | Activation transfer across models · IHY and Code Vulnerability tasks | reconstruction and language-modeling losses | Mixed |
| Replace target activations with mapped activations instead of mean-ablated activations | Qwen 1.5B and Llama 3B transfer validations · IHY and Code Vulnerability datasets | LLM-Judge: 0.0 → 2.6; 0.9 → 4.5 | Helped |
| Reduce mapped steering-vector magnitude below 5 | Qwen 1.5B Code Vulnerability backdoor transfer · Code Vulnerability backdoor removal | backdoor steering success rate | Helped |
| Use mapped refusal steering rather than the native target refusal vector | Llama 3B · Refusal, The Pile | perplexity: 15.78 → 16.50 | Hurt |
| Use mapped refusal steering rather than the native target refusal vector | Llama 3B · Refusal, Alpaca | perplexity: 7.26 → 8.32 | Hurt |
| Transfer the multi-layer humor steering vector with the autoencoder mapping | Target model · Humorous response generation | tested prompts successfully steered: 60% | Helped |
| Use activation patching instead of weight patching | Qwen 0.5B backdoored model · Base-to-fine-tuned behavior switching | layers required for comparable backdoor mitigation: approximately 50% | Helped |
| Add activations from additional layers to the transferred steering intervention | Llama corrupted-capabilities models · Corrupted capabilities | steering success rate: 6.34% | Helped |
Surprising findings
- Transferred steering from Gemma 2B to Llama 3B performed poorly, suggesting tokenizer/vocabulary differences hinder cross-architecture transfer.
- Refusal vector mapped from Llama 1B to 3B produces completions starting with refusal phrases that are then followed by harmful content, so substring matching underestimates jailbreaks while Llama-Guard scores look comparable.
Limitations
- Steering sweeps to pick layers are expensive, and the approach operates on a single layer for most tasks.
- Cross-architecture transfer struggles when vocabularies differ significantly (e.g., Gemma to Llama); tested on limited model sizes.
- MMLU performance drops drastically for mapped IHY models, while instruction following is better preserved.
Experimental setup
Datasets: hh-rlhf WildGuardMix harmful and harmless dataset The Pile Alpaca GPT-4o-mini generated Astralisia facts
Benchmarks: JailbreakBench MMLU AlpacaEval subset
Metrics: LLM-judge score (0-5) coherence score KL-divergence substring match Llama-Guard safe fraction perplexity ROUGE BERTScore BLEURT backdoor trigger rate
Baselines: affine map native target steering vector mean ablation LoRA
Builds on: Activation Addition difference-in-means refusal direction sleeper agents model stitching
Closest papers by meaning
- Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-Based Decision-Making Systems (ICLR 2025 · similarity 0.87)
- Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.86)
- Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.86)
- Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors (NeurIPS 2026 · similarity 0.86)
- Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection (CVPR 2026 · similarity 0.86)
- Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.