Research map / Architectures and efficiency / Attention and transformers
On the Emergence of Position Bias in Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Graph-theoretic analysis of how causal, sliding-window and prefix masks, decay masks and RoPE shape position bias across multi-layer attention, with synthetic retrieval experiments.
Problem
How attention masks and positional encodings shape position bias (e.g., lost-in-the-middle, attention sinks) in multi-layer transformers, which lacks a comprehensive theoretical understanding.
Approach
Model attention masks as directed graphs and interpret multi-layer attention rollout probabilities as path-sum distributions; show causal masks make contextual mass converge exponentially to early (center) tokens, and that decay masks and RoPE induce per-layer distance decay which, across depth, trades off against cumulative early-token importance.
Key results
- Causal mask theorem: contextual probability to the first token converges to 1 exponentially with depth, with explicit rate bounds.
- Decay mask yields P(z_i=j) = Theta(C(t+i-j, i-j) e^{-(i-j)m}), so the critical distance x* = t/(e^m-1) grows with depth and shrinks with decay strength.
- Experiments show deeper attention amplifies bias toward earlier tokens; decay mask induces stronger recency bias than RoPE; causal mask without PE captures bias only at the first position.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase attention depth without residual connections | single-head attention network · synthetic retrieval with position-unbiased training data | position bias toward earlier tokens | Helped |
| add a decay mask | causal attention · synthetic retrieval with position-unbiased training data | focus on recent tokens | Helped |
| add RoPE | causal attention · synthetic retrieval with position-unbiased training data | focus on recent tokens | Helped |
| use a decay mask instead of RoPE | causal attention · synthetic retrieval with position-unbiased training data | focus on recent tokens | Helped |
| add standard residual connections after each attention layer (ablation) | single-head attention network · synthetic retrieval with position-unbiased training data | position bias by depth | Mixed |
| use a causal mask instead of no mask, without positional encoding | attention network with residual connections · synthetic retrieval trained with bias toward the first and last positions | position bias toward the first position | Helped |
| add sinusoidal positional encoding | attention network with or without a causal mask · synthetic retrieval trained with bias toward the first and last positions | capture of positional biases at both ends | Helped |
| add RoPE | attention network with or without a causal mask · synthetic retrieval trained with bias toward the first and last positions | capture of positional biases at both ends | Helped |
| train with positional bias toward both the beginning and the end | attention network with positional encodings · synthetic retrieval | lost-in-the-middle pattern | Helped |
| use a causal mask without positional encoding instead of no mask without positional encoding | attention network with residual connections · synthetic retrieval trained with positional bias at the beginning or end | capture of positional bias at the last position | Hurt |
| use sinusoidal positional encoding or RoPE instead of no positional encoding | attention network · synthetic retrieval trained with positional bias at the middle or last position | capture of positional bias at the trained position | Helped |
| use a smaller sliding-window width | sliding-window attention · multi-layer attention | bias toward early tokens | Helped |
Surprising findings
- Causal mask without positional encoding captures positional bias only toward the beginning of the sequence, not the end, contrary to the hypothesis that it simulates absolute or relative PE.
- Residual connections make the depth-bias relationship non-monotonic, sometimes amplifying and sometimes reducing bias depending on PE and depth.
Limitations
- Theory assumes bounded query/key/value weights (A1-A2) and, for RoPE, only slowly rotating dimensions (effective d=2) in the main analysis.
- Experiments use a simplified attention-only synthetic setup rather than real LLMs.
Experimental setup
Datasets: synthetic Gaussian mixture retrieval sequences
Benchmarks: synthetic retrieval test sets with positional gap [a,b]-[b,a]
Metrics: retrieval accuracy gap attention sink metric
Baselines: no mask no PE sinusoidal PE
Builds on: attention rollout ALiBi RoPE Birdie-style in-context learning data generation
Closest papers by meaning
- What are you sinking? A geometric approach on attention sink (NeurIPS 2025 · similarity 0.92)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.91)
- Consensus Is All You Get: The Role of Attention in Transformers (ICML 2025 · similarity 0.89)
- Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps (ICLR 2024 · similarity 0.89)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.