Research map / Architectures and efficiency / Attention and transformers
A Primal-Dual Framework for Transformers and Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add batch normalization to attention keys/queries (Attention-BN) | softmax transformer · UEA time series (10 datasets) | average test accuracy (%): +0.82 (72.07 to 72.89) | Helped |
| add scaled heads (Attention-SH) | softmax transformer · UEA time series (10 datasets) | average test accuracy (%): +0.75 (72.07 to 72.82) | Helped |
| combine Attention-BN and Attention-SH | softmax transformer · UEA time series (10 datasets) | average test accuracy (%): +1.15 over baseline | Helped |
| add batch normalization (Attention-BN) | softmax transformer · LRA (5 tasks) | average test accuracy (%): +0.62 (58.66 to 59.28) | Helped |
| add scaled heads (Attention-SH) | softmax transformer · LRA (5 tasks) | average test accuracy (%): +0.52 (58.66 to 59.18) | Helped |
| add batch normalization (Attention-BN) | DeiT-tiny · ImageNet | top-1 accuracy (%): +0.56 (72.23 to 72.79) | Helped |
| add scaled heads (Attention-SH) | DeiT-tiny · ImageNet | top-1 accuracy (%): -0.15 (72.23 to 72.08) | No clear effect |
| add scaled heads (Attention-SH) to reduce head redundancy (ablation) | softmax transformer on LRA retrieval · LRA Retrieval | mean L2 distance between heads: 2.01 to 3.81 | Helped |
| add batch normalization to attention on LRA Retrieval for head diversity (ablation) | softmax transformer on LRA retrieval · LRA Retrieval | mean L2 distance between heads: 2.01 to 2.45 | Helped |
| learn beta instead of hyperparameter in Attention-BN (ablation) | Attention-BN on LRA retrieval · LRA Retrieval | test accuracy (%): 80.77 (learned) vs 81.05 (hyperparameter) | No clear effect |
| add Attention-SH and BN to Sparse Attention baseline | sparse attention · UEA time series subset | average test accuracy (%): Attention-BN+SH 72.88 vs baseline 71.98 | Helped |
| add Attention-Conv1D on LRA (ablation) | softmax transformer · LRA (5 tasks) | average test accuracy (%): +0.35 (58.66 to 59.01) | Helped |
Closest papers by meaning
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.91)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.90)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.89)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.88)
- Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (ICLR 2024 · similarity 0.88)
- Memorization Capacity of Multi-Head Attention in Transformers (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.