Research map / Architectures and efficiency / Attention and transformers

Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes inference-time mean-field transformer token dynamics at low temperature, proving Wasserstein concentration onto the projection of the initial token distribution onto the dominant eigenspace of VB^T, with a quantitative bound in beta and time.

Problem

Describing how tokens evolve through deep encoder-only self-attention layers in the large-token, infinite-depth limit, and quantifying how the token distribution concentrates at low temperature (large beta) over inference time.

Approach

Couples the transformer continuity equation to its zero-temperature limit via a quantitative Laplace principle and a Wasserstein stability estimate, and uses Lyapunov functionals to show exponential convergence of the zero-temperature flow to the push-forward of the initial datum under a projection onto the dominant eigenspace of VB^T.

Key results

• Theorem gives W_2(rho_t^beta, Pi_# rho_0) <= 2 sqrt(log(beta+1)/beta)(e^{C_1 t}-e^{C_0 t}) + V_p(rho_0) exp(-p gamma t / sigma_max(B)), so concentration holds for times up to order log beta.
• Corollary: the zero-temperature flow converges exponentially in W_2 to Pi_# rho_0 with rate governed by the spectral gap gamma.
• Numerical experiments show that for finite beta and large t, tokens enter a terminal phase aligning with the dominant eigenspace of V rather than VB^T.

Surprising findings

• For finite beta and large times, the dynamics leave the eigenspace of VB^T and concentrate on the dominant eigenspace of V, a terminal phase not captured by the zero-temperature limit.

Limitations

• Assumes symmetric VB^T and invertible B, restricting the dynamics compared to the general case.
• Analysis is for infinite depth and infinitely many tokens with a single head and no MLPs; finite layers, finite tokens, multi-head attention not covered.
• Assumption on the initial density (positive Lipschitz density) is stronger than needed for the zero-temperature result; the terminal phase is only numerically supported.

Experimental setup

Metrics: Wasserstein distance alignment with dominant eigenspaces

Builds on: mean-field transformer continuity equation consensus-based optimization quantitative Laplace principle

Closest papers by meaning

• A multiscale analysis of mean-field transformers in the moderate interaction regime (NeurIPS 2025 · similarity 0.93)
• Emergence of meta-stable clustering in mean-field transformer models (ICLR 2025 · similarity 0.89)
• Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers (NeurIPS 2026 · similarity 0.88)
• Universal Approximation of Mean-Field Models via Transformers (ICML 2025 · similarity 0.86)
• Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.86)
• Local to Global: Learning Dynamics and Effect of Initialization for Transformers (NeurIPS 2024 · similarity 0.86)