Research map / Architectures and efficiency / Attention and transformers

The mechanistic basis of data dependence and abrupt learning in an in-context classification task

ICLR 2024 · Architectures and efficiency › Attention and transformers · approach: Training dynamics of in-context learning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that in a minimal two-layer attention-only network trained on a Gaussian-mixture ICL/IWL classification task, ICL arises from an abruptly formed induction head, and a phenomenological three-nested-logit model explains the abrupt transition.

Problem

What data distributional and architectural factors favor in-context versus in-weights learning, and what mechanism produces the abrupt emergence of in-context learning in transformers?

Approach

Reproduces data-dependence results (burstiness, number of classes, within-class variability, Zipf exponent) in a minimal attention-only network on Gaussian-mixture inputs; identifies induction-head formation as the driver of the abrupt transition via progress measures and ablations; and explains the abruptness with a phenomenological loss of three nested logits learned sequentially, producing clif

Key results

• The minimal two-layer attention-only network reproduces the burstiness, class-number, within-class variability and Zipf dependencies of ICL vs IWL reported in prior work.
• IC accuracy shows a slow learning phase followed by an abrupt transition coinciding with induction-head progress measures; the two-parameter induction-head model with beta2=0 reproduces the full data-distribution dependencies.
• Setting L=N removes the slow phase and only 2 of 6 seeds acquire the ICL solution, while the phenomenological three-logit model explains this via a saddle at the origin.

Measured effects reported in the paper

Surprising findings

• ICL is not transient in these small-network simulations, contrasting with prior reports of transient ICL in larger 12-layer models.
• When L=N (no slow phase), four of six seeds converge to a sub-optimal minimum instead of learning ICL.

Limitations

• Minimal toy setup may use different mechanisms than large LLMs.
• Phenomenological model ignores stochasticity and finite embedding dimension D.
• Heuristics such as weight tying used in LLM training were not included.

Experimental setup

Datasets: synthetic Gaussian mixture item-label sequences Omniglot (referenced in prior work)

Benchmarks: synthetic in-context and in-weights accuracy on controlled Gaussian-mixture tasks

Metrics: in-weights accuracy in-context accuracy swapped-label accuracy progress measures (ILA1, TILA2, CLA, TLA2) loss curves

Baselines: 12-layer transformer from prior ICL/IWL study

Builds on: induction heads Chan et al. data distributional properties of ICL

Closest papers by meaning

• Toward Understanding In-context vs. In-weight Learning (ICLR 2025 · similarity 0.91)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers (NeurIPS 2024 · similarity 0.91)
• Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.89)
• Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.89)
• Probing the Decision Boundaries of In-context Learning in Large Language Models (NeurIPS 2024 · similarity 0.89)
• Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence (ICML 2025 · similarity 0.89)