Research map / Architectures and efficiency / Attention and transformers
The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a bilinear-form framework showing that autoregressive training induces column-dominant directionality and bidirectional training induces symmetry in the query-key matrix W_qk of self-attention, validates this across pretrained models and modalities, and shows symmetric initialization speeds up encoder-only training.
Problem
Understanding how information is embedded in self-attention weight matrices and how autoregressive versus bidirectional training objectives shape their structure remains unclear.
Approach
Reformulates self-attention through the bilinear form W_qk = W_q W_k^T, shows the implicit gradient update of W_qk is a sum of rank-1 terms weighted by context/prediction counts, and derives that autoregressive objectives yield column dominance and directionality while bidirectional objectives yield approximate symmetry; uses this to motivate symmetric initialization.
Key results
- Pretrained encoder-only models show median symmetry scores of about 0.4-0.8 (e.g. BERT-tiny 0.77) versus near 0 for many decoder-only models (e.g. LLaMA3-8B 0.00).
- Symmetric initialization reaches the non-symmetric 4-layer Wikipedia model's final loss up to 73% faster (loss 0.812 vs 0.984 final).
- Symmetric initialization gives lower final loss for most configurations, e.g. 24-layer Wikipedia 0.166 vs 0.192.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use symmetric W_qk initialization instead of independent initialization (ablation) | 4-layer encoder-only Transformer · Jigsaw; Wikipedia; Red Pajama | loss: 2.782 -> 2.758; 0.984 -> 0.812; 1.106 -> 0.907 | Helped |
| Use symmetric W_qk initialization instead of independent initialization (ablation) | 12-layer encoder-only Transformer · Jigsaw; Wikipedia; Red Pajama | loss: 1.419 -> 1.430; 0.256 -> 0.247; 0.297 -> 0.274 | Mixed |
| Use symmetric W_qk initialization instead of independent initialization (ablation) | 24-layer encoder-only Transformer · Jigsaw; Wikipedia; Red Pajama | loss: 0.786 -> 0.739; 0.192 -> 0.166; 0.209 -> 0.189 | Helped |
| Use symmetric W_qk initialization instead of independent initialization (ablation) | 4-layer encoder-only Transformer · Jigsaw; Wikipedia; Red Pajama | speed-up: 26 %; 73 %; 69 % | Helped |
| Use symmetric W_qk initialization instead of independent initialization (ablation) | 12-layer encoder-only Transformer · Jigsaw; Wikipedia; Red Pajama | speed-up: 0 %; 20 %; 35 % | Mixed |
| Use symmetric W_qk initialization instead of independent initialization (ablation) | 24-layer encoder-only Transformer · Jigsaw; Wikipedia; Red Pajama | speed-up: 14%; 44%; 34% | Helped |
| Use symmetric W_qk initialization on vision Transformers (ablation) | Vision Transformers · Vision tasks | performance | No clear effect |
Surprising findings
- Symmetric initialization speed-up is not significant for vision transformers.
- Decoder-only models show weak symmetry even at initialization-independent late training stages, with skew-symmetric structure in middle layers.
Experimental setup
Datasets: Wikipedia Jigsaw Red Pajama
Benchmarks: Wikipedia Jigsaw Red Pajama
Metrics: symmetry score directionality score final training loss training speed-up
Baselines: random initialization
Builds on: Transformer self-attention masked language modeling (BERT) logit lens
Closest papers by meaning
- From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics (NeurIPS 2025 · similarity 0.91)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.90)
- What are you sinking? A geometric approach on attention sink (NeurIPS 2025 · similarity 0.89)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.89)
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.88)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.