Research map / Architectures and efficiency / Attention and transformers
The emergence of sparse attention: impact of data distribution and benefits of repetition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how sparse attention emerges during training in a single-location linear regression toy model and a small Transformer on associative recall, showing emergence time follows power laws in data properties and that repetition accelerates it.
Problem
Understanding how and when emergent abilities appear during training, and why they appear suddenly, by linking emergence to the formation of sparse attention and to how data distribution (sequence length, dimension, repetition) affects emergence timing.
Approach
Uses a linear regression variant requiring attention to a single token as an analytically tractable model of sparse attention emergence; derives ODE dynamics and plateau-escape time, shows in-context and cross-sample repetition speed emergence, and checks predictions on small Transformers and an associative recall (induction head) task.
Key results
- Emergence (plateau) time in the toy model follows T_plateau = 1.51 T^0.99 B^-0.99 d^0.49 (R^2=0.999) for in-context repetition.
- On a 2-layer 4-head Transformer, plateau length follows 0.76 d^1.29 T^0.80 B^-0.80 (R^2=0.995).
- On associative recall, repetition speeds emergence by about 4x for in-context repetition and 2x for cross-sample repetition, with overfitting at large repetition.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Repeat the relevant token B times within the context (ablation) | Simplified Transformer · Single-location linear regression | plateau length: T_\mathrm{plateau} = 1.51 \, T^{\,0.99} \, B^{-0.99} \, d^{\,0.49} | Helped |
| Repeat the relevant token across samples with probability p (ablation) | Simplified Transformer · Single-location linear regression | plateau length: T_\mathrm{plateau} = 2.15 (\sqrt{d}T / \sqrt{p^2d + (1- p)^2})^{1.02} | Helped |
| Increase sequence length T (ablation) | Simplified Transformer · Single-location linear regression | plateau length: T_\varepsilon = \frac{\sqrt{d}T}{2} \, \ln\left ( \varepsilon \sqrt{d}T \right ) | Hurt |
| Increase input/output dimension d (ablation) | Simplified Transformer · Single-location linear regression | plateau length: T_\varepsilon = \frac{\sqrt{d}T}{2} \, \ln\left ( \varepsilon \sqrt{d}T \right ) | Hurt |
| Replace Adam with SGD (ablation) | 2-layer 4-head Transformer · Single-location linear regression | emergence speed | Hurt |
| Increase number of pairs in the context (ablation) | Transformer · Associative recall | plateau length: T_\mathrm{plateau} = 0.55 \, N_\mathrm{tokens}^{0.79}\, N_\mathrm{pairs}^{2.25} | Hurt |
| Increase vocabulary size (ablation) | Transformer · Associative recall | plateau length: T_\mathrm{plateau} = 0.55 \, N_\mathrm{tokens}^{0.79}\, N_\mathrm{pairs}^{2.25} | Hurt |
| Repeat the query as a key within the context (ablation) | Transformer · Associative recall | emergence speed: by a factor of 4 | Helped |
| Use too much in-context repetition (ablation) | Transformer · Associative recall | test accuracy | Hurt |
| Increase cross-sample repetition probability p (ablation) | Transformer · Associative recall | emergence speed: by a factor of 2 | Helped |
| Use too much cross-sample repetition (ablation) | Transformer · Associative recall | test accuracy | Hurt |
| Increase cross-sample repetition probability p (ablation) | Transformer · Single-location linear regression | test loss | Mixed |
Surprising findings
- Repetition of the relevant token speeds emergence even when evaluated on non-repeated data, but can cause overfitting and only a temporary advantage.
- Replacing Adam with SGD substantially slows emergence in Transformers.
Limitations
- Theory relies on gradient flow, uniform attention initialization, and large T and d assumptions; the toy model parameterizes attention scores directly without semantic or positional information.
- Cross-sample repetition gives only a temporary advantage and causes overfitting in realistic architectures.
- Power-law exponents differ across optimizer, architecture and task, and the high N_pairs exponent is only hypothesized.
Experimental setup
Datasets: single-location linear regression (synthetic) in-context associative recall (synthetic)
Benchmarks: single-location linear regression in-context associative recall
Metrics: plateau length test loss accuracy
Baselines: SGD
Builds on: induction head deep linear networks Adam
Closest papers by meaning
- Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence (NeurIPS 2026 · similarity 0.90)
- A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language (ICLR 2025 · similarity 0.89)
- An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem (NeurIPS 2024 · similarity 0.89)
- What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformers (NeurIPS 2025 · similarity 0.88)
- A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention (NeurIPS 2024 · similarity 0.88)
- What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.