AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Why Transformers Struggle with Distribution-Independent In-Context Learning

Omar Naim, Jerome Bolte, Nicholas Asher

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Training dynamics of in-context learning · 8 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Tests whether transformers' in-context learning implements general implicit learning algorithms by evaluating them on controlled regression tasks beyond the training distribution.

Problem

Whether transformer in-context learning reflects implicit learning algorithms that generalize beyond the training task distribution, in controlled regression settings where classical estimators are available.

Approach

Evaluates ICL outside the training distribution in controlled regression settings and compares transformer predictions against classical statistical estimators, rather than assuming the transformer implements a general learning algorithm.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
scale coefficients beyond training range (sigma 1 -> 10)Transformer ICL · Degree-1 polynomial regression, coefficient shiftMSE: 9×10^-6 -> 27.06Hurt
replace transformer with kernel ridge regression (poly kernel)KRR (Poly) · Degree-1 polynomial regression, coefficient shiftMSE at sigma=10: 27.06 -> 2.3×10^-3Improved
remove final normalization at test time only (ablation)Transformer · Degree-1 polynomial regression, coefficient shiftMSE at sigma=1: 9×10^-6 -> 434Hurt
train from scratch without final normalization (ablation)Transformer · Degree-1 polynomial regression, coefficient shiftMSE at sigma=10: 27.06 -> 35.98No clear effect
attention-only 2-layer architecture (ablation)Transformer · Degree-1 polynomial regression, coefficient shiftMSE at sigma=10: 27.06 -> 39.17No clear effect
flatten attention with temperature scaling or alternative attention (sparsemax, entmax, linear) (ablation)Transformer · Degree-1 polynomial regression, coefficient shiftMSENo clear effect
basis-aligned structured training (T1) (ablation)Transformer · Degree-1 polynomial regression, coefficient shiftMSE at sigma=1: 9×10^-6 -> 0.11Hurt
multi-directional structured training (T2) vs T1 (ablation)Transformer · Degree-1 polynomial regression, coefficient shiftMSE at sigma=1: 0.11 -> 0.01Improved
train for more steps (ablation)Transformer · Degree-1, in-distributionMSENo clear effect
expand training support to U(-5,5) or U(-10,10) (ablation)Transformer · Degree-1 polynomial regression, coefficient shiftMSEImproved
expand training support to U(-100,100) (ablation)Transformer · Degree-1 polynomial regression, coefficient shiftMSEHurt
increase polynomial degree from 1 to 6 (ablation)Transformer · Polynomial regression, sigma=10MSE: 27.1 -> 37.7Mixed

Experimental setup

Baselines: classical estimators

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy