Research map / Architectures and efficiency / Attention and transformers

Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Attention head mechanisms · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives a closed-form Bayesian posterior over the attention matrix of a single-head attention network trained on a copy task, reducing it to two order parameters that predict a second-order-then-crossover transition for linear attention and a first-order transition for softmax attention in training-set size.

Problem

Explaining why the copy subcircuit (first layer of an induction head) emerges abruptly during training, and whether the emergence is a first-order, second-order, or crossover phenomenon as a function of training data amount.

Approach

Marginalize embedding and attention-weight matrices to get a Gaussian-process prior and data likelihood over the attention matrix, then reduce the posterior via permutation symmetry to two order parameters (uniform pooling and copy) whose action's topology determines transition order; softmax vs linear activation changes the order.

Key results

• Softmax attention shows a sharp first-order transition near P≈300 with a discontinuous loss drop, with theory predicting P*~L ln^2 L, an O(sqrt(LV)) improvement over linear attention.
• Linear attention shows a second-order transition at P≈100 (predicted ~23 in one listing) followed by a crossover near P≈322-400 toward copy attention.
• Theory predictions for order parameters agree with models trained with Adam and sampled with SGLD.

Surprising findings

• Linear attention's later copy reorganization is a crossover rather than a second phase transition because the Z2 symmetry is already broken at the first transition.

Limitations

• Single-layer, single-head setting with supervised copy task and no causal masking, not full induction heads with next-token prediction.
• Relies on equivalent-kernel approximation and d_model to infinity limit, suppressing finite-size corrections.
• Assumes i.i.d. uniform token distribution; structured or natural-language distributions not studied.

Experimental setup

Datasets: synthetic copy task (i.i.d. one-hot tokens)

Benchmarks: synthetic copy task

Metrics: loss per token order parameters (c1, cG, a1, aG)

Baselines: linear attention

Builds on: mu-P scaling NNGP correspondence for attention equivalent-kernel approximation Adam SGLD

Closest papers by meaning

• The emergence of sparse attention: impact of data distribution and benefits of repetition (NeurIPS 2025 · similarity 0.90)
• What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation (ICML 2024 · similarity 0.89)
• A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention (NeurIPS 2024 · similarity 0.88)
• Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers (NeurIPS 2024 · similarity 0.88)
• Selective Induction Heads: How Transformers Select Causal Structures in Context (ICLR 2025 · similarity 0.87)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention (ICML 2025 · similarity 0.87)