Research map / Architectures and efficiency / Attention and transformers
Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that for linearized-attention transformers, in-context learning demonstrations can be exactly converted into permanent key-value bias terms (ICLCA), with an approximate variant for softmax transformers evaluated on GPT-2.
Problem
In-context learning demonstrations must be re-supplied in every prompt and do not permanently change model weights; existing context distillation methods are empirical, approximate, and require expensive gradient-based fine-tuning.
Approach
In linearized attention, the key-value matrix aggregates ICL token information linearly, so adding a bias term to the key-value matrix (and normalizer) reproduces the effect of ICL prompt tokens exactly, with the bias computed by a cheap forward pass; for softmax models, the same idea is applied approximately via a kernel (Performer-style) approximation.
Key results
- Exact conversion for linear attention models from 205K to 1.98B parameters yields relative logit errors of 2.9e-7 to 4.3e-6.
- On the induction head task, the converted model reaches 99.95% in-context accuracy, matching the model with ICL prompt, versus 2.23% without the ICL prompt.
- On pretrained GPT-2 with approximate (Performer-style) conversion, relative logit error to the with-ICL model drops from 16.56% to 9.17%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Convert the ICL prompt into bias terms with ICLCA | Linear attention transformer · Induction head task | in-context accuracy: 2.23% -> 99.95% | Helped |
| Approximate ICL conversion with bias terms | GPT-2 · GPT-2 logits with ICL prompt | relative error: 16.56% -> 9.17% | Helped |
| Add bias terms to the linearized attention module (ablation) | Linear attention transformer trained from scratch · Induction head task | training speed | Helped |
Surprising findings
- Direct conversion of ICL prompts into existing weights for linear attention is not generally possible; an added bias term is required.
Limitations
- Conversion is exact only for linearized attention; for softmax GPT-2 it is approximate with substantial logit error (9.17%).
- Experiments are small-scale: synthetic induction-head task and GPT-2 qualitative examples, without downstream benchmarks.
- Requires assumptions such as autoregressive attention, relative positional encoding, and separable normalizer.
Experimental setup
Datasets: synthetic induction head bigram data
Benchmarks: synthetic induction head task GPT-2 logit relative error on text prompts
Metrics: relative error of logits in-context accuracy
Baselines: context distillation (fine-tuning-based) M_old with ICL prompt M_old without ICL prompt
Builds on: linear attention (Katharopoulos et al.) Performer RoFormer rotary embeddings induction head task (Bietti et al.)
Closest papers by meaning
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.92)
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.91)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.91)
- In-Context Deep Learning via Transformer Models (ICML 2025 · similarity 0.90)
- Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens (NeurIPS 2024 · similarity 0.90)
- Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.