Research map / Architectures and efficiency / Attention and transformers
A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes a tied low-rank dot-product attention model in the high-dimensional limit, giving a closed-form characterization of its global empirical-loss minimum and a positional-to-semantic phase transition in sample complexity.
Problem
Theoretically understanding how attention layers learn positional versus semantic mechanisms and when sharp transitions between them emerge with increasing sample complexity.
Approach
Uses a solvable model of single-layer tied low-rank dot-product attention on Gaussian tokens, maps ERM to a GLM and derives tight finite-dimensional self-consistent equations (via GAMP / replica) for the global minimum, revealing a sharp transition from positional to semantic attention as the number of samples grows.
Key results
- Positional minimum is the global minimum for sample complexity alpha below a threshold alpha_c, and the semantic minimum is global above it, giving a sharp phase transition.
- Dot-product attention outperforms the purely positional linear baseline above a threshold alpha_l that is at least alpha_c in all probed settings.
- Theoretical predictions for overlaps and test MSE agree well with gradient descent experiments at finite size, with agreement improving as n,d grow.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase sample complexity so the learned mechanism transitions from positional to semantic attention | Dot-product attention · Gaussian two-token attention task | test MSE | Helped |
| Use dot-product attention instead of a dense linear positional model at sufficient sample complexity | Attention model versus dense linear baseline · Gaussian two-token attention task | test MSE | Helped |
| Use dot-product attention instead of a dense linear positional model below the semantic-learning threshold | Attention model versus dense linear baseline · Gaussian two-token attention task | test MSE | Hurt |
| Continue training the positional or semantic solution after unfreezing parameters | Histogram-task transformer solutions · Histogram task test set | accuracy | Helped |
| Continue training the positional or semantic solution after unfreezing parameters | Histogram-task transformer solutions · Histogram task | training loss | Helped |
| Restrict attention to positional information rather than semantic information (ablation) | Single-layer transformer on the histogram task · Histogram task test set | accuracy: 0.99885 ± 0.0004; 0.99895 ± 0.0012 | No clear effect |
| Restrict attention to semantic information rather than positional information (ablation) | Single-layer transformer on the histogram task · Histogram task test set | accuracy: 0.99895 ± 0.0012; 0.99885 ± 0.0004 | No clear effect |
Surprising findings
- Positional attention solution is outperformed by a linear positional baseline in the positional regime, so the dot-product model is worse than a simple linear model until semantic learning occurs.
Limitations
- Analysis characterizes only minima of the loss landscape, not gradient descent dynamics from random initialization.
- Relies on the assumed point-wise convergence of GAMP and the high-dimensional limit; rigorous justification left open.
- Simplified architecture: tied low-rank query/key, identity value, single head and layer, Gaussian 1-gram data.
Experimental setup
Datasets: synthetic Gaussian sentences (L=2 tokens) with teacher attention histogram task (synthetic)
Metrics: test mean squared error training loss summary statistics overlap
Baselines: linear positional baseline
Builds on: generalized approximate message passing state evolution replica method
Closest papers by meaning
- Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning (NeurIPS 2025 · similarity 0.88)
- Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence (NeurIPS 2026 · similarity 0.88)
- The emergence of sparse attention: impact of data distribution and benefits of repetition (NeurIPS 2025 · similarity 0.88)
- From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers (ICML 2024 · similarity 0.88)
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.88)
- Understanding Emergent Abilities of Language Models from the Loss Perspective (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.