Research map / Architectures and efficiency / Attention and transformers
Toward Understanding In-context vs. In-weight Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a theoretical model (gated in-weight vs in-context predictors with regret and generalization analysis) explaining why in-context learning emerges and can become transient, validated on synthetic transformers, Omniglot, and fine-tuning Gemini Nano 1.
Problem
Explain when in-context learning (ICL) emerges in transformers under distributional properties of training data and why it can diminish (become transient) with further training, as opposed to in-weight learning (IWL).
Approach
A bi-level gating model selects between a tabular in-weight predictor and a softmax-attention-style in-context predictor per input, choosing whichever has lower expected test error; generalization bounds show IW error decays with sample count while IC error has a floor set by irrelevant context labels, so ICL is transient when IWL becomes more accurate. A no-regret bi-level update learns this sele
Key results
- On synthetic data, the transformer's ICL on relevant contexts diminishes as N grows once the IW predictor's error falls below the IC predictor's, with ICL persisting for sigma=0.4 input noise.
- On Omniglot, ICL emerges on low-frequency classes and becomes transient as N increases even without input noise.
- Fine-tuning Gemini Nano 1 (1.8B) on eight person-city facts causes the model to revert to the memorized city in some cases where the context states an imaginary city, reducing ICL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase training-set size | Transformer on synthetic classification · Synthetic classification, low-frequency classes | 0-1 validation error | Mixed |
| Increase input noise (ablation) | Transformer on synthetic classification · Synthetic classification | 0-1 validation error | Helped |
| Increase the number of low-frequency classes (ablation) | Transformer on synthetic classification · Synthetic classification, low-frequency classes | 0-1 validation error | Mixed |
| Use a balanced class distribution (ablation) | Transformer on synthetic classification · Synthetic classification | 0-1 validation error | Mixed |
| Increase label noise (ablation) | Transformer on synthetic classification · Synthetic classification | 0-1 validation error | Helped |
| Increase the probability of relevant contexts (ablation) | Transformer on synthetic classification · Synthetic classification | 0-1 validation error | Mixed |
| Reduce the number of relevant contexts (ablation) | Transformer on synthetic classification · Synthetic classification, context length 4 | 0-1 validation error | Hurt |
| Increase context length (ablation) | IC predictor on synthetic classification · Synthetic classification | 0-1 validation error | Hurt |
| Fine-tune to memorize name–city pairs | Gemini Nano 1 · Natural-language prompts with a conflicting in-context city | selection of the in-context answer | Hurt |
| Fine-tune to memorize name–city pairs | Gemini Nano 1 · Natural-language prompts with a conflicting in-context city | probability of selecting the memorized answer | Hurt |
| Train on Omniglot with many low-frequency classes | Transformer on Omniglot · Omniglot, low-frequency classes | 0-1 validation error | Mixed |
Surprising findings
- Increasing the number of low-frequency classes makes ICL harder to get off the ground but makes it diminish more slowly.
- With too few relevant contexts (L_relevant=1 of L=4) the transformer never learns ICL.
Limitations
- Theory uses a stylized tabular in-weight predictor and simplified softmax-based in-context predictor; full transformer mechanics are not analyzed.
- LLM experiment is a small qualitative demonstration with eight finetuning questions on a proprietary model.
- Some transience phenomena depend on finite-time performance, which is hard to analyze theoretically.
Experimental setup
Datasets: synthetic Gaussian-prototype classification data Omniglot custom person-city finetuning questions
Benchmarks: synthetic classification (IBD/OOBD 0-1 error) Omniglot (IBD/OOBD 0-1 error)
Metrics: 0-1 error cross-entropy regret
Baselines: IC predictor IW predictor stylized ICL model GRU
Builds on: induction heads Chan et al. 2022 data distributional properties of ICL Krichevsky-Trofimov estimator online-to-batch conversion
Closest papers by meaning
- The mechanistic basis of data dependence and abrupt learning in an in-context classification task (ICLR 2024 · similarity 0.91)
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.90)
- Breaking through the learning plateaus of in-context learning in Transformer (ICML 2024 · similarity 0.90)
- In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.90)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.90)
- Dual Operating Modes of In-Context Learning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.