Research map / Architectures and efficiency / Attention and transformers
The Transient Nature of Emergent In-Context Learning in Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| continue training after ICL first emerges | Transformer trained on bursty Omniglot data · Omniglot, default settings | ICL evaluator accuracy | Hurt |
| continue training after ICL first emerges | Transformer trained on bursty Omniglot data · Omniglot, default settings | IWL evaluator accuracy | Helped |
| increase model depth (ablation) | Transformer trained on Omniglot · Omniglot, varying model depth | ICL evaluator accuracy | No clear effect |
| increase exemplars per class (ablation) | Transformer trained on Omniglot · Omniglot, varying in-class variation | ICL evaluator accuracy | Mixed |
| increase the number of classes (ablation) | Transformer trained on Omniglot · Omniglot, varying class count | ICL evaluator accuracy | Helped |
| increase class count rather than in-class variation at fixed dataset size (ablation) | Transformer trained on Omniglot · Omniglot, matched total exemplar-label pairs | ICL evaluator accuracy | Helped |
| replace image exemplars with clustered LLaMa token embeddings (ablation) | Transformer trained on exemplar-label data · LLaMa token-embedding datasets | ICL evaluator accuracy | Mixed |
| use a moderate Zipfian skew (ablation) | Transformer trained on Omniglot · Omniglot, Zipf exponent 1 | ICL evaluator accuracy | Helped |
| use an extreme Zipfian skew (ablation) | Transformer trained on Omniglot · Omniglot, Zipf exponent 2 | ICL evaluator accuracy | Hurt |
| add L2 regularization (ablation) | Transformer trained on Omniglot · Omniglot, regularization experiments | ICL evaluator accuracy | Helped |
| increase L2 regularization beyond the useful range (ablation) | Transformer trained on Omniglot · Omniglot, extreme regularization | ICL evaluator accuracy: >0.0001 | Hurt |
| increase embedding dimension (ablation) | Transformer trained on Omniglot · Omniglot, varying embedding dimension | ICL evaluator accuracy | Helped |
Closest papers by meaning
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.91)
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.91)
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.91)
- Breaking through the learning plateaus of in-context learning in Transformer (ICML 2024 · similarity 0.90)
- Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens (NeurIPS 2024 · similarity 0.90)
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.