Research map / Architectures and efficiency / Attention and transformers

Clustering in Causal Attention Masking

NeurIPS 2024 · Architectures and efficiency › Attention and transformers · approach: Theory of self-attention dynamics · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Modifies self-attention particle dynamics on the sphere to causal masking, proves asymptotic collapse to a single cluster for V=I with arbitrary Q,K, and connects metastable clustering centers to Rényi parking.

Problem

Existing theory of transformer clustering as interacting particle systems covers only full (unmasked) self-attention, whereas practical generative transformers use causal masking; the paper asks whether causal attention dynamics collapse and form metastable clusters.

Approach

Model causally masked self-attention as an interacting particle system on the sphere in which token k attends only to tokens j ≤ k; this loses the gradient-flow structure, so convergence is proven via a sequential/hierarchical argument, and metastable cluster centers are linked to Rényi parking.

Key results

• For V = I_d and arbitrary Q, K, causal attention dynamics converge almost surely to a single cluster (all tokens tend to x_1(0)).
• Conjectured atlas of final configurations depends only on the top eigenvalue and eigenspace of V, with Q, K and beta not mattering.
• Strong Rényi centers stay quasi-stationary for time exponential in squared separation; with frozen centers, all particles converge to them under conditions such as beta >= 14 and N <= 700 for eps=0.1, c=6.5.

Surprising findings

• Pre-trained albert-xlarge-v2 has some value heads with negative largest eigenvalue, unlike Gaussian initialization.

Limitations

• Metastable clustering is not rigorously proven; no bounds on convergence time, so strong Rényi centers may not stay stationary during aggregation.
• Tied weights across layers and the MLP layer are omitted.
• General V with non-identity eigenstructure is only conjectured.

Experimental setup

Metrics: asymptotic cluster configuration fraction of particles consumed by Rényi centers

Baselines: full (mean-field) self-attention dynamics of Geshkovski et al.

Builds on: Geshkovski et al. particle view of self-attention Lojasiewicz gradient flow convergence center-stable manifold theorem

Closest papers by meaning

• Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers (NeurIPS 2026 · similarity 0.88)
• On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.87)
• Scaling Limits of Long-Context Transformers (NeurIPS 2026 · similarity 0.86)
• Bayes optimal learning of attention-indexed models (NeurIPS 2025 · similarity 0.86)
• Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.86)
• High-Dimensional Learning Dynamics of Attention-Indexed Models (NeurIPS 2026 · similarity 0.86)