Research map / Architectures and efficiency / Attention and transformers

Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies how training an FFN-only noisy mean-field Transformer reshapes token clustering, identifying an initial clustering, turnpike plateau, and terminal escape phase via an entropy-regularized interaction energy and turnpike/adjoint analysis.

Problem

Existing mean-field analyses of Transformer dynamics treat weights as prescribed, leaving open how training reshapes token clustering across layers.

Approach

Treats the layerwise FFN parameters as optimal-control variables in a noisy symmetric unnormalized self-attention mean-field model with L2 regularization, showing that training can induce a turnpike plateau followed by terminal escape from the clustered regime, analyzed via an entropy-regularized interaction energy with double-well landscape.</account>

Key results

• Exponential turnpike bound: energy gap Delta(mu_t*) <= C(e^{-at}+e^{-a(T-t)}) for optimal controls and large T.
• One-step GD adjoint analysis shows energy gap can grow exponentially near the terminal layer, with rate given by a Rayleigh quotient of the loss gradient.
• Numerical particle experiments show smaller lambda_2 yields a more persistent plateau and sharper terminal escape; fitted rates match theoretical Rayleigh quotient trends.

Limitations

• Analysis restricted to noisy S-USA attention with FFN-only training and linear-in-parameters FFN.
• Only small-scale particle simulations (N=64, d=2); no validation on real Transformers.
• Relies on assumptions (non-degenerate positive attention matrix, energy-threshold conditions, local controllability).

Experimental setup

Metrics: signed angle interaction energy fitted exponential decay rate

Baselines: prescribed-weight mean-field clustering (W=0) one-step gradient descent

Builds on: geshkovski2025perspective mean-field clustering Esteve-Yague turnpike bootstrap neural ODE optimal control

Closest papers by meaning

• A multiscale analysis of mean-field transformers in the moderate interaction regime (NeurIPS 2025 · similarity 0.90)
• Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape (ICML 2024 · similarity 0.88)
• Emergence of meta-stable clustering in mean-field transformer models (ICLR 2025 · similarity 0.88)
• JoMA: Demystifying Multilayer Transformers via Joint Dynamics of MLP and Attention (ICLR 2024 · similarity 0.88)
• Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime (NeurIPS 2026 · similarity 0.88)
• Clustering in Causal Attention Masking (NeurIPS 2024 · similarity 0.88)