Research map / Architectures and efficiency / Attention and transformers

Differential learning kinetics govern the transition from memorization to generalization during in-context learning

ICLR 2025 · Architectures and efficiency › Attention and transformers · approach: Training dynamics of in-context learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows via a one-layer transformer and a minimal additive model that the memorization-to-generalization (IWL-to-ICL) transition in in-context learning is governed by differential learning rates of memorization vs. generalization sub-circuits, with a memorization scaling law explaining the task diversity threshold.

Problem

Understanding why in-context learning (ICL) emerges sharply versus in-weights memorization as task diversity increases, and whether this is due to limited network capacity or relative learning dynamics.

Approach

Models the one-layer transformer as two independent sub-circuits (an MLP for memorization and an attention head for ICL) whose logits add; the transition depends on the relative rates at which they learn, not on memorization capacity, yielding exponential sensitivity to initialization, a memorization scaling law I_K ~ K^nu, and the power law K* ~ N^{1/nu}.

Key results

• The task diversity threshold K* scales with context length N with exponent about 1.41, closely matching the predicted 1/nu ≈ 1.43 from the memorization scaling law I_K(∞) ~ K^0.7.
• Time to acquire ICL scales linearly with context length N in the K→∞ limit, with long-tailed distributions across ~100 seeds.
• Near the transition, solutions are bimodal across ~250 seeds, and enforcing equal numbers of +1 and -1 labels in context substantially reduces the probability of acquiring ICL.

Measured effects reported in the paper

Surprising findings

• Whether ICL is acquired depends on initial parameters with exponential sensitivity (t_ICL ~ exp(-beta_0)), producing long-tailed acquisition times.
• ICL is transient under L2 regularization on the attention head because slow memorization of the MLP eventually lets regularization decay the ICL weight.

Limitations

• The minimal model relies on ansatzes (independence and interaction strength) and is phenomenological rather than derived from the transformer.
• Results are for a small one-layer synthetic setting; extension to larger models and naturalistic tasks is untested.
• Theory assumes D to infinity and K much greater than N; breakdown at finite D for large N is not characterized.

Experimental setup

Datasets: synthetic in-context classification dataset (Gaussian items, random binary labels)

Metrics: ICL accuracy IWL accuracy time to ICL acquisition ICL loss vs IWL loss

Baselines: capacity-constrained hypothesis

Builds on: Chan et al. ICL data distributional properties

Closest papers by meaning

• When can in-context learning generalize out of task distribution? (ICML 2025 · similarity 0.93)
• In-Context Learning Strategies Emerge Rationally (NeurIPS 2025 · similarity 0.90)
• Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.89)
• In-Context Learning through the Bayesian Prism (ICLR 2024 · similarity 0.89)
• Strategy Coopetition Explains the Emergence and Transience of In-Context Learning (ICML 2025 · similarity 0.89)
• Can In-context Learning Really Generalize to Out-of-distribution Tasks? (ICLR 2025 · similarity 0.89)