Research map / Architectures and efficiency / Attention and transformers

A multiscale analysis of mean-field transformers in the moderate interaction regime

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes the continuous-depth mean-field self-attention dynamics on the sphere in the moderate interaction regime (beta growing with N), identifying three timescale phases (alignment, heat, pairing) and their limiting equations.

Problem

Describing how tokens evolve through the depth of encoder-only transformers at inference time when the inverse temperature beta scales with the number of tokens N, a regime used in practice (e.g., SSMax, YaRN), which lacks a rigorous multiscale description.

Approach

Models tokens as a mean-field interacting particle system on the sphere (with layer normalization) and, as beta grows, derives three timescale-separated limiting dynamics: a linear transport equation for collapse onto a low-dimensional eigen-subspace (alignment), a forward/backward heat equation within it (heat), and ODEs for sequential cluster merging along geodesics (pairing).

Key results

• Alignment phase: under invertible Q,K,V, the empirical token measure converges to a linear transport limit and collapses onto the eigenspace of VK^TQ with largest real part (N=10^4 simulations on S^2, beta=30).
• Heat phase: when Q^TK restricted to the subspace equals lambda_1 I and V=±lambda_2 I, the rescaled dynamics approach a forward or backward heat equation, with simulations matching analytical solutions (beta=10 backward, beta=50 forward on S^1).
• Pairing phase: the closest pair of clusters merges along their geodesic on an exponential timescale exp(beta(1-<x_i,x_j>)), proved in arbitrary dimension.

Surprising findings

• The alignment-phase limit is a linear transport equation, and the paper notes the token dynamics can keep rotating indefinitely (e.g., V a rotation with Q^TK=Id), so no stationary limit exists in general.

Limitations

• Backward heat phase is ill-posed and only local-in-time solutions exist beyond the first collapse; full rigorous treatment remains open.
• MLP layers are excluded from the analysed dynamics.
• Heat-phase assumptions (Q^TK positive definite on the subspace, V=±lambda I) are restrictive; stability of E_max under finite beta is not proven.

Experimental setup

Metrics: empirical vs analytical distribution agreement

Builds on: Sander et al. 2022 sinkformers / continuous transformer dynamics neural ODEs diffusion-velocity methods

Closest papers by meaning

• Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime (NeurIPS 2026 · similarity 0.93)
• Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers (NeurIPS 2026 · similarity 0.90)
• Emergence of meta-stable clustering in mean-field transformer models (ICLR 2025 · similarity 0.89)
• Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.88)
• Infinite Limits of Multi-head Transformer Dynamics (NeurIPS 2024 · similarity 0.87)
• Universal Approximation of Mean-Field Models via Transformers (ICML 2025 · similarity 0.86)