Research map / Architectures and efficiency / Attention and transformers
Emergence of meta-stable clustering in mean-field transformer models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies the intermediate-time meta-stable clustering of tokens in a mean-field continuous-time transformer (Q=K=V=Id, unit sphere), proving linear, quasi-linear and clustering phases and showing the emerging periodicity is set by the Gegenbauer index maximizing a rescaled coefficient sequence.
Problem
Long-time analyses of token clustering in mean-field transformer dynamics show total collapse to one cluster, but do not explain the intermediate meta-stable partial-clustering phases seen numerically, which matter for finite-depth representations.
Approach
Perturbative analysis of the mean-field PDE around the iid uniform initialization on the sphere: the perturbation grows along the Gegenbauer mode k_max maximizing gamma_k = k(k+d-2) W_k, giving k_max-periodic structure that persists for O(ln N) time, with Dobrushin-type propagation of chaos on finite intervals.
Key results
- Numerical experiments with beta=5 and beta=7 yield k_max=3 and 4, matching the number of clusters observed in simulations of 10^4 particles on S^1.
- Exit time from the quasi-linear regime grows logarithmically in N (N from 1000 to 16000, 20 runs per N), consistent with the O(ln N) meta-stable duration prediction.
- Dobrushin estimate gives W1(mu_t,nu_t) <= e^{2Ct} W1(mu_0,nu_0); a two-particle counterexample shows the exponential time dependence cannot be improved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase inverse temperature β from 5 to 7 | Mean-field Transformer dynamics · S^1, 10^4 particles | number of clusters during the meta-stable phase: 3 -> 4 | Helped |
| Initialize tokens independently and uniformly on S^1 | Mean-field Transformer dynamics · S^1 | number of clusters during the meta-stable phase: β=5: 3; β=7: 4 | Helped |
| Increase the number of particles N | Mean-field Transformer dynamics · β=2; N=1000 to N=16000 | exit time from the quasi-linear regime | Mixed |
Surprising findings
- A two-particle counterexample shows the exponential-in-time growth of the Dobrushin stability bound cannot generally be improved, so uniform-in-time propagation of chaos fails for this system.
Limitations
- Analysis restricted to Q=K=V=Id, no MLP layers, and d=2 for the precise statements; higher-dimensional case only sketched.
- Assumption that the limit after the quasi-linear phase is a superposition of k_max delta masses is not proven and is assumed.
- Propagation of chaos holds only on finite (or O(log N)) time intervals.
Experimental setup
Metrics: number of clusters exit time from quasi-linear regime vs N Wasserstein/total variation distance
Builds on: Geshkovski et al. (2023) mean-field transformer model Dobrushin stability estimate Grenier's iterative scheme
Closest papers by meaning
- A multiscale analysis of mean-field transformers in the moderate interaction regime (NeurIPS 2025 · similarity 0.89)
- Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime (NeurIPS 2026 · similarity 0.89)
- Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers (NeurIPS 2026 · similarity 0.88)
- Universal Approximation of Mean-Field Models via Transformers (ICML 2025 · similarity 0.87)
- Clustering in Causal Attention Masking (NeurIPS 2024 · similarity 0.83)
- Global Convergence in Training Large-Scale Transformers (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.