Research map / Architectures and efficiency / Attention and transformers

Capturing In-Context Learning Dynamics with Task Operators

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Training dynamics of in-context learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Task Operator, a training-free method that extracts per-head affine parameters of ICL attention outputs and replays them as an updated output projection at zero-shot inference, outperforming prior ICL-compression baselines.

Problem

ICL requires processing all demonstrations at every inference, and prior methods that compress ICL into fixed activation vectors fail on tasks needing fine-grained input-dependent computation.

Approach

Shows each attention head's ICL output is an exact affine transform (scale w plus context bias b) of its context-masked output, and replays these parameters as a dynamic update to the attention output projection at zero-shot inference, rather than fixed additive vectors.

Key results

• On Llama3.1-8B, TO reaches 75.82% on GSM8K vs 80.14% for 8-shot ICL and 37.91% zero-shot, beating TV (41.32%), FV, ICV and Conceptor.
• On Qwen3-4B Reverse, TO gets 39.06% vs 43.75% ICL and at most 1.56% for baselines' TV/FV/Conceptor, showing large gains on algorithmic tasks.
• Batch-averaged TO (TO batch) shows non-decreasing accuracy with K and can exceed ICL on Deduplicate at K=64 (Llama3.1-8B).

Surprising findings

• Scaling-only or bias-only operators at template positions collapse to zero accuracy on non-reasoning tasks, while the joint affine form works.
• Sparsifying to a validation-selected subset of sites improves Reverse and Deduplicate accuracy by up to 9.4 pp on some models.

Limitations

• Performance becomes less stable at very large K as attention mass on non-context positions shrinks, increasing variance.
• Evaluation limited to text-only decoder LLMs on lexical, algorithmic and reasoning tasks.
• Sparsification requires a validation sweep and may overfit; the dense operator is used in main results.

Experimental setup

Datasets: Translation word pairs (Kaggle-style word-pair data cited) Linguistic word pairs synthetic Uppercase/Reverse/Deduplicate GSM8K train MATH train GPQA-Main

Benchmarks: Translation Linguistic Uppercase Reverse Deduplicate GSM8K MATH500 GPQA-Diamond

Metrics: exact match accuracy

Baselines: Task Vectors Function Vectors In-Context Vectors Conceptor zero-shot 8-shot ICL

Builds on: function vectors task vectors in-context vectors conceptors chain-of-thought

Closest papers by meaning

• Revisiting In-context Learning Inference Circuit in Large Language Models (ICLR 2025 · similarity 0.92)
• DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning (NeurIPS 2024 · similarity 0.91)
• Implicit In-context Learning (ICLR 2025 · similarity 0.91)
• Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.91)
• In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.90)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers (NeurIPS 2024 · similarity 0.90)