Research map / Architectures and efficiency / Attention and transformers
Why Transformers Struggle with Distribution-Independent In-Context Learning
Omar Naim, Jerome Bolte, Nicholas Asher
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tests whether transformers' in-context learning implements general implicit learning algorithms by evaluating them on controlled regression tasks beyond the training distribution.
Problem
Whether transformer in-context learning reflects implicit learning algorithms that generalize beyond the training task distribution, in controlled regression settings where classical estimators are available.
Approach
Evaluates ICL outside the training distribution in controlled regression settings and compares transformer predictions against classical statistical estimators, rather than assuming the transformer implements a general learning algorithm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale coefficients beyond training range (sigma 1 -> 10) | Transformer ICL · Degree-1 polynomial regression, coefficient shift | MSE: 9×10^-6 -> 27.06 | Hurt |
| replace transformer with kernel ridge regression (poly kernel) | KRR (Poly) · Degree-1 polynomial regression, coefficient shift | MSE at sigma=10: 27.06 -> 2.3×10^-3 | Improved |
| remove final normalization at test time only (ablation) | Transformer · Degree-1 polynomial regression, coefficient shift | MSE at sigma=1: 9×10^-6 -> 434 | Hurt |
| train from scratch without final normalization (ablation) | Transformer · Degree-1 polynomial regression, coefficient shift | MSE at sigma=10: 27.06 -> 35.98 | No clear effect |
| attention-only 2-layer architecture (ablation) | Transformer · Degree-1 polynomial regression, coefficient shift | MSE at sigma=10: 27.06 -> 39.17 | No clear effect |
| flatten attention with temperature scaling or alternative attention (sparsemax, entmax, linear) (ablation) | Transformer · Degree-1 polynomial regression, coefficient shift | MSE | No clear effect |
| basis-aligned structured training (T1) (ablation) | Transformer · Degree-1 polynomial regression, coefficient shift | MSE at sigma=1: 9×10^-6 -> 0.11 | Hurt |
| multi-directional structured training (T2) vs T1 (ablation) | Transformer · Degree-1 polynomial regression, coefficient shift | MSE at sigma=1: 0.11 -> 0.01 | Improved |
| train for more steps (ablation) | Transformer · Degree-1, in-distribution | MSE | No clear effect |
| expand training support to U(-5,5) or U(-10,10) (ablation) | Transformer · Degree-1 polynomial regression, coefficient shift | MSE | Improved |
| expand training support to U(-100,100) (ablation) | Transformer · Degree-1 polynomial regression, coefficient shift | MSE | Hurt |
| increase polynomial degree from 1 to 6 (ablation) | Transformer · Polynomial regression, sigma=10 | MSE: 27.1 -> 37.7 | Mixed |
Experimental setup
Baselines: classical estimators
Closest papers by meaning
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.93)
- How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (ICLR 2024 · similarity 0.93)
- Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens (NeurIPS 2024 · similarity 0.92)
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.92)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.92)
- In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
