Research map / Architectures and efficiency / Attention and transformers
Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes the continuous-time limit of self-attention to give conditions under which token embeddings converge to zero or diverge, shows RoPE promotes divergence, and proposes learnable regularization of the query-key matrix that gives small perplexity/BPC gains.
Problem
Understanding how tokens evolve through pre-trained Transformer self-attention layers (treated as a continuous-time dynamical system), and how absolute and rotary positional encodings affect whether tokens converge or diverge, with prior theory relying on restrictive assumptions.
Approach
Treats the residual self-attention update as an ODE with matrices A = W(V^T)^{-1} and W, proves token-distance monotonicity and convergence-to-zero or divergence conditions under broader assumptions than prior work, then uses these regimes to constrain attention and to add a learnable negative-definite term to RoPE's query-key matrix to discourage convergence.
Key results
- Under the divergence constraint on WikiText-103 the validation perplexity is 31.12 versus 31.63 baseline and 32.24 under convergence (sinusoidal PE).
- On DeiT ImageNet-1K, divergence constraint gives top-1 71.96% versus 71.80% baseline and 71.34% under convergence.
- With RoPE, adding a learnable term lambda*A lowers EnWik8 test BPC from 1.295 to 1.281 and WikiText-103 test perplexity from 32.35 to 32.04.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Constrain attention parameters to the convergence scenario | Transformer with sinusoidal positional encoding · EnWik8 test | BPC: 1.331 -> 1.350 | Hurt |
| Constrain attention parameters to the divergence scenario | Transformer with sinusoidal positional encoding · EnWik8 test | BPC: 1.331 -> 1.324 | Helped |
| Constrain attention parameters to the convergence scenario | Transformer with sinusoidal positional encoding · WikiText-103 validation | PPL: 31.63 -> 32.24 | Hurt |
| Constrain attention parameters to the divergence scenario | Transformer with sinusoidal positional encoding · WikiText-103 validation | PPL: 31.63 -> 31.12 | Helped |
| Constrain attention parameters to the convergence scenario | DeiT · ImageNet-1K validation | Top 1 Acc: 71.80 -> 71.34 | Hurt |
| Constrain attention parameters to the divergence scenario | DeiT · ImageNet-1K validation | Top 1 Acc: 71.80 -> 71.96 | Helped |
| Add learnable regularization term λI_D to W_li (ablation) | Transformer + RoPE · EnWik8 test | BPC: 1.295 -> 1.288 | Helped |
| Add learnable regularization term λA to W_li | Transformer + RoPE · EnWik8 test | BPC: 1.295 -> 1.281 | Helped |
| Add learnable regularization term λA to W_li | Transformer + RoPE · WikiText-103 validation | PPL: 31.37 -> 31.06 | Helped |
Surprising findings
- Absolute positional encoding has minimal effect on token dynamics, while RoPE promotes divergence.
- Gains from the proposed RoPE modification are modest (e.g., about 0.3 perplexity points).
Limitations
- Analysis omits layer normalization and feed-forward networks, and the full-model case is only supported empirically.
- Convergence and divergence sufficiency beyond special cases (e.g., A symmetric, V with positive eigenvalues) are supported only by random simulations rather than proofs.
- Performance gains from the proposed modifications are modest.
Experimental setup
Datasets: WikiText-103 EnWik8 ImageNet-1K
Benchmarks: WikiText-103 EnWik8 ImageNet-1K
Metrics: perplexity bits per character top-1 accuracy top-5 accuracy
Baselines: Transformer with sinusoidal positional encoding Transformer + RoPE DeiT
Builds on: Geshkovski et al. particle-system analysis of self-attention ResNet continuous-time limit RoPE ALBERT
Closest papers by meaning
- On the Emergence of Position Bias in Transformers (ICML 2025 · similarity 0.91)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.90)
- Consensus Is All You Get: The Role of Attention in Transformers (ICML 2025 · similarity 0.89)
- Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers (NeurIPS 2025 · similarity 0.89)
- Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities (NeurIPS 2025 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.