Research map / Architectures and efficiency / Attention and transformers
What are you sinking? A geometric approach on attention sink
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a geometric "reference frame" account of attention sinks, classifying them as centralized, distributed, or bidirectional, and links these types to position encoding choices using topology, spectral graph, KL, and Fisher analyses on pretrained models.
Problem
Attention sinks, where certain tokens attract disproportionate attention regardless of semantics, are usually treated as architectural artifacts or model-specific quirks; the paper asks what geometric function they serve and how architecture shapes them.
Approach
Attention sinks are reframed as reference frames that anchor representation manifolds. Three types (centralized, distributed, bidirectional) are tied to position encoding: standard RoPE gives centralized, NTK-scaled RoPE gives distributed, and absolute embeddings give bidirectional. Support comes from analyses of existing pretrained models, not from a training-time intervention.
Key results
- Centralized (Llama-3.2-3B) shows 100% attention specialization on the BOS token with constant Betti_0 of 26.43 and zero Betti_1 across layers.
- Distributed (Qwen2.5-7B) shows a sign flip in Fiedler vs. centralization correlation from -0.46 at low threshold to +0.61 at high threshold.
- Bidirectional (XLM-RoBERTa) shows Betti_0 dropping from 22.68 to 1.69 and Betti_1 from 19.69 to 3.11 across layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NTK-aware scaled RoPE instead of standard RoPE | Decoder-only transformer · STEM Wikipedia sentences | Avg. sink concentration: 82.93% -> 69.92% | Mixed |
| Use NTK-aware scaled RoPE instead of standard RoPE | Decoder-only transformer · STEM Wikipedia sentences | Avg. KL Reduction (t=0.8): -0.0974 -> -0.0088 | Mixed |
| Use NTK-aware scaled RoPE instead of standard RoPE | Decoder-only transformer · STEM Wikipedia sentences | High-threshold algebraic connectivity effectiveness: 0.04 (1/28 layers) -> 0.25 (7/28 layers) | Helped |
| Use absolute position embeddings instead of standard RoPE | Transformer · STEM Wikipedia sentences | Final layer Betti_0: 26.43 -> 1.69 | Mixed |
| Use absolute position embeddings instead of standard RoPE | Transformer · STEM Wikipedia sentences | High-threshold algebraic connectivity effectiveness: 0.04 (1/28 layers) -> 0.38 (9/24 layers) | Helped |
| Use absolute position embeddings instead of standard RoPE | Transformer · STEM Wikipedia sentences | Avg. sink concentration: 82.93% -> 66.86% | Mixed |
Surprising findings
- Attention sinks in the centralized model show negative geometric-semantic alignment, which the authors read as geometric rather than semantic organization.
- The Llama-3.2-3B spectral correlation signs flip across thresholds, which is used as a signature of the frame type.
Limitations
- Topological and spectral analyses use snapshots at specific layers rather than tracking training dynamics.
- Claims that reference frames emerge early in training rest on RMT analysis of Pythia checkpoints, with limited evidence.
- Causal claims about position encoding determining frame type are supported only by correlational analysis across a small set of models.
Experimental setup
Datasets: STEM-focused Wikipedia sentences (6-50 tokens)
Metrics: Betti numbers persistence Fiedler value star-likeness degree centralization KL reduction Fisher information norm directional influence geometric-semantic alignment
Builds on: RoPE ALiBi Pythia
Closest papers by meaning
- On the Emergence of Position Bias in Transformers (ICML 2025 · similarity 0.92)
- Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention (NeurIPS 2026 · similarity 0.90)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.89)
- Consensus Is All You Get: The Role of Attention in Transformers (ICML 2025 · similarity 0.89)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.