Research map / Architectures and efficiency / Attention and transformers
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
Viet-Hoang Tran, Khanh Vinh Bui, Van-Hoan Trinh, Tan Lai Ngoc, Tan Minh Nguyen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Stage 2 (orthogonal alignment) on top of Stage 1 head permutation (ablation) | weight matching for attention · CIFAR-10 / CIFAR-100 / IMDB / DBPedia, APE and RoPE | loss barrier ratio (%): 78.3 -> 10.2 (CIFAR-10 APE) | Improved |
| add gradient descent fine-tuning to Stage 2 orthogonal initialization (ablation) | weight matching for attention · CIFAR-10 / CIFAR-100 / IMDB / DBPedia, APE and RoPE | loss barrier ratio (%): 10-16% -> 7-12% | Improved |
| add gradient descent to Stage 2 (Variant 2 to full) (ablation) | weight matching for attention · CIFAR-10 (APE) | loss barrier ratio (%): 10.2 -> 8.7 | Improved |
| add gradient descent to Stage 2 (Variant 2 to full) (ablation) | weight matching for attention · IMDB reviews (APE) | accuracy barrier ratio (%): 15.3 -> 10.1 | Improved |
| use Stage 1 head permutation matching vs best of 24 permutations | weight matching for attention · CIFAR-10 (APE) | rank out of 24 (loss): 2.40 | No clear effect |
| re-initialize the entire Transformer on large-scale benchmarks | attention LMC · ImageNet, WikiText-103, Enwik8, One Billion Word | linear mode connectivity | Hurt |
| re-initialize only the first attention layer | attention LMC · vision and text benchmarks | linear mode connectivity | Improved |
Closest papers by meaning
- On the Emergence of Position Bias in Transformers (ICML 2025 · similarity 0.90)
- What are you sinking? A geometric approach on attention sink (NeurIPS 2025 · similarity 0.89)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.88)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- Positional Attention: Expressivity and Learnability of Algorithmic Computation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
