Research map / Architectures and efficiency / Attention and transformers

Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretically and empirically studies attention sinks versus diagonal attention as mechanisms for attention switch and oversmoothing prevention in single-head causal transformers, showing sinks are necessary for hard switches and cheaper to represent than diagonal patterns for large context.

Problem

Explaining why attention sinks emerge: how sinks are geometrically represented, whether dense attention causes oversmoothing, whether sinks are the only way to switch attention off, and why sinks are preferred over diagonal (self-attention) patterns in pretrained transformers.

Approach

Characterizes representability of sinks via half-space conditions on embeddings, derives conditions under which denser attention increases cosine-similarity oversmoothing including skip connections and value-output maps, proves sinks are the only hard attention switch under mild geometry, and compares l2/nuclear-norm costs of sink versus diagonal patterns, showing sinks are cheaper for large conte

Key results

• Empirically, the approximation of average token cosine similarity closely matches measured values across LLaMA3-8B, Gemma-7B, GPT2-XL and Mistral-7B; roughly half of heads satisfy the condition under which denser attention increases oversmoothing, while few satisfy the decreasing condition.
• BOS cosine similarity with all other embeddings does not cross zero in middle layers of gpt2-small, Llama 3.1-8B and Gemma 7B, consistent with sink representability.
• Theory shows sink cost scales ~T while diagonal cost scales ~T^{3/2} for the backcopy task, so sinks are cheaper at large context length; diagonal attention appears in 6%, 0.6% and 9.5% of heads of gpt2-medium, Llama 3.1-8B and Gemma 7B.

Surprising findings

• Denser attention does not by itself increase oversmoothing; skip connections and value-output maps can counteract mixing, though this is rare in trained LLMs.
• Under mild geometric conditions, sinks are the only possible mechanism for a hard attention switch in optimal transformers.

Limitations

• Single-head and single-layer analysis; multi-head oversmoothing progression not characterized.
• Oversmoothing analysis relies on high-dimensional approximation and assumed token distribution (Z_i = z-bar + epsilon_i) and uniform-ish single-step attention.
• Conjectured lower bound on sink cost of order r_eta + C is not proven; sink-vs-diagonal comparisons rely on synthetic distributions.

Experimental setup

Datasets: TinyStories tinyshakespeare WikiText CodeSearchNet-Python C4

Benchmarks: C4

Metrics: average pairwise cosine similarity cosine similarity between BOS and other embeddings head-level attention mass classification parameter cost (l2 / nuclear norm)

Baselines: attention sink theory of Barbero et al. diagonal attention

Builds on: attention sinks (Xiao et al.) bigram-backcopy task active-dormant mechanism

Closest papers by meaning

• What are you sinking? A geometric approach on attention sink (NeurIPS 2025 · similarity 0.90)
• On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
• Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.87)
• Spectral Conditioning of Attention Improves Transformer Performance (NeurIPS 2025 · similarity 0.87)
• On the Emergence of Position Bias in Transformers (ICML 2025 · similarity 0.87)
• Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers (NeurIPS 2026 · similarity 0.86)