Research map / Large language models / Language, safety and interpretability
Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insights
Haolin Yang, Hakaze Cho, Kaize Ding, Naoya Inoue
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the task vector by gradient descent instead of extracting it | task vector injection · 7-dataset avg, layer 16 last token | accuracy (%): 37.80 (Vanilla TV), 37.30 (FV) -> 83.49 | Improved |
| inject LTV at a different token position | LTV · 7-dataset avg | accuracy (%): 83.49 -> 78.39 | No clear effect |
| inject at multiple positions (ablation) | LTV · 7-dataset avg | accuracy (%): 83.49 -> 86.43 | Improved |
| inject at multiple positions (ablation) | Function Vector · 7-dataset avg | accuracy (%): 37.30 -> 31.88 | Hurt |
| inject at many layers and positions simultaneously (ablation) | LTV · 7-dataset avg | accuracy (%): 83.49 -> 51.39 | Hurt |
| inject LTV into 8-shot ICL prompts | LTV · 7-dataset avg | accuracy (%): ICL slightly below 80 -> 84.61 | Improved |
| use LTV instead of Prefix Tuning or LoRA as PEFT | Llama3.1-8B · SST-2 | accuracy (%): 85.67 (Prefix), 91.63 (LoRA) -> 92.89 | Improved |
| reconstruct TV effect via attention-head OV circuits (ablation) | LTV · 7-dataset avg | accuracy | Improved |
| ablate top-10% key heads vs random 10% heads (ablation) | LTV · 7-dataset avg | accuracy | Hurt |
| apply SNLI-trained LTV to other tasks | LTV · cross-task | accuracy (%): 46.21 on RTE, 0.00 on SST-2/TREC | Mixed |
Closest papers by meaning
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.93)
- Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning (NeurIPS 2026 · similarity 0.93)
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.92)
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.91)
- Where does In-context Learning Happen in Large Language Models? (NeurIPS 2024 · similarity 0.90)
- Vector-ICL: In-context Learning with Continuous Vector Representations (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
