Research map / Architectures and efficiency / Attention and transformers
Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses an encoder-decoder view of in-context learning to show that task encoding (separable latent-task representations) and task decoding emerge together during pretraining, and that a kNN-based task decodability score predicts ICL accuracy in LLMs.
Problem
How task vectors form during pretraining and why their effectiveness for in-context learning varies across tasks.
Approach
Frames ICL as task encoding (mapping latent tasks to separable representations) plus task decoding (conditional algorithms), introduces Task Decodability (kNN accuracy on intermediate representations) as a geometric proxy that predicts ICL performance, and shows finetuning early layers improves encoding more than finetuning late layers.
Key results
- On Llama-3.1-8B, TD scores correlate positively with 4-shot ICL accuracy across POS and bitwise arithmetic tasks.
- Finetuning the first 10 layers outperforms finetuning the last 10 layers by 37% on POS tagging and 24% on bitwise arithmetic at 4 shots (Table: FT first 10 avg acc 69.4 vs 33.9 for POS; 90.2 vs 66.3 for bitwise).
- Prompting raises average TD for bitwise arithmetic from 0.43 to 0.90 and accuracy from 26.2 to 57.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Finetune the first 10 layers instead of using the pretrained model | Llama-3.1-8B · POS tagging | Avg. Acc.: 25.5 { $\pm$ 6.7} -> 69.4 { $\pm$ 14.2} | Helped |
| Finetune the first 10 layers instead of using the pretrained model | Llama-3.1-8B · POS tagging | Avg. TD: 0.68 { $\pm$ 0.11} -> 0.95 { $\pm$ 0.02} | Helped |
| Finetune the first 10 layers instead of using the pretrained model | Llama-3.1-8B · Bitwise arithmetic | Avg. Acc.: 26.2 { $\pm$ 15.5} -> 90.2 { $\pm$ 15.0} | Helped |
| Finetune the first 10 layers instead of using the pretrained model | Llama-3.1-8B · Bitwise arithmetic | Avg. TD: 0.43 { $\pm$ 0.13} -> 0.85 { $\pm$ 0.11} | Helped |
| Finetune the last 10 layers instead of using the pretrained model | Llama-3.1-8B · POS tagging | Avg. TD: 0.68 { $\pm$ 0.11} -> 0.68 { $\pm$ 0.11} | No clear effect |
| Finetune the last 10 layers instead of using the pretrained model | Llama-3.1-8B · Bitwise arithmetic | Avg. TD: 0.43 { $\pm$ 0.13} -> 0.43 { $\pm$ 0.13} | No clear effect |
| Finetune the last 10 layers instead of using the pretrained model | Llama-3.1-8B · POS tagging | Avg. Acc.: 25.5 { $\pm$ 6.7} -> 33.9 { $\pm$ 12.3} | Helped |
| Finetune the last 10 layers instead of using the pretrained model | Llama-3.1-8B · Bitwise arithmetic | Avg. Acc.: 26.2 { $\pm$ 15.5} -> 66.3 { $\pm$ 9.01} | Helped |
| Provide the true task concept in the prompt instead of using the pretrained prompt | Llama-3.1-8B · POS tagging | Avg. Acc.: 25.5 { $\pm$ 6.7} -> 34.4 { $\pm$ 10.3} | Helped |
| Provide the true task concept in the prompt instead of using the pretrained prompt | Llama-3.1-8B · Bitwise arithmetic | Avg. Acc.: 26.2 { $\pm$ 15.5} -> 57.2 { $\pm$ 30.1} | Helped |
| Provide the true task concept in the prompt instead of using the pretrained prompt | Llama-3.1-8B · POS tagging | Avg. TD: 0.68 { $\pm$ 0.11} -> 0.97 { $\pm$ 0.02} | Helped |
| Provide the true task concept in the prompt instead of using the pretrained prompt | Llama-3.1-8B · Bitwise arithmetic | Avg. TD: 0.43 { $\pm$ 0.13} -> 0.90 { $\pm$ 0.10} | Helped |
Surprising findings
- Finetuning earlier layers improves ICL more than finetuning the last layers, contrary to common practice.
- Finetuning the last 10 layers leaves task decodability unchanged from the pretrained model.
Limitations
- Experiments do not cover tasks requiring multi-step reasoning.
- TD metric is only meaningful when computed over semantically similar, confusable tasks.
- Natural-task analysis relies on constructed tasks with small test sets (e.g., 500 bitwise samples) and a single pretrained model in several analyses.
Experimental setup
Datasets: Penn Treebank (POS tagging) synthetic sparse linear regression bitwise arithmetic (constructed)
Benchmarks: POS tagging (constructed from Penn Treebank) bitwise arithmetic (constructed) synthetic sparse linear regression
Metrics: exact-match accuracy task decodability (kNN accuracy) MSE
Baselines: Prompting FT (last 10 layers) FT (first 10 layers)
Builds on: task vectors (Hendel et al.) Bayesian view of in-context learning activation patching LoRA
Closest papers by meaning
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.91)
- Does learning the right latent variables necessarily improve in-context learning? (ICML 2025 · similarity 0.91)
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.90)
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.90)
- Scaling Sparse Feature Circuits For Studying In-Context Learning (ICML 2025 · similarity 0.90)
- Dual Operating Modes of In-Context Learning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.