Research map / Architectures and efficiency / Attention and transformers
Improving Transformer with an Admixture of Attention Heads
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace 8-head softmax with 4 global-head GFiSHformer | Transformer · WikiText-103 | test PPL: 34.29 -> 33.71 | Helped |
| use 2 global heads instead of 8 heads (FiSHformer) | Transformer · WikiText-103 | test PPL: 34.29 -> 34.96 | Hurt |
| use 2 global heads GFiSHformer instead of 8 heads | Transformer · WikiText-103 | test PPL: 34.29 -> 34.48 | No clear effect |
| mixture of heads instead of admixture of heads (ablation) | FiSHformer 2 global heads · WikiText-103 | test PPL: 34.96 -> 36.28 | Hurt |
| vary number of global heads in GFiSHformer (2,4,6) (ablation) | GFiSHformer · WikiText-103 | test PPL: 34.48 / 33.71 / 33.80 | Mixed |
| add learnable noise (non-hard) FiSHformer vs Hard (ablation) | GFiSHformer 2 global heads · IWSLT14 De-En | BLEU: 34.54 -> 34.71 | Helped |
| apply FiSH to linear transformer | Linear 4 heads · IWSLT14 De-En | BLEU: 28.22 -> 30.24 | Helped |
| replace 16 heads with 8 global-head FiSHformer | Softmax 16 heads · WMT14 En-De | BLEU: 29.38 -> 29.57 | Helped |
| replace 16 heads with 4 global-head Hard GFiSHformer | Softmax 16 heads · WMT14 En-De | BLEU: 29.38 -> 28.97 | Hurt |
| use GFiSH Swin-T with 6/12 global heads instead of 12/24 heads | Swin-T · ImageNet-1k | top-1 accuracy: 81.20 -> 81.11 | No clear effect |
| compare FiSHformer to adaptively sparse transformer | Transformer · WMT14 En-De | BLEU: 26.93 -> 27.67 | Helped |
| use GFiSHformer instead of softmax | Transformer · UEA time series | average accuracy: 72.07 -> 72.17 | No clear effect |
Closest papers by meaning
- Memorization Capacity of Multi-Head Attention in Transformers (ICLR 2024 · similarity 0.88)
- SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024 · similarity 0.88)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.88)
- MemoryFormer : Minimize Transformer Computation by Removing Fully-Connected Layers (NeurIPS 2024 · similarity 0.87)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.87)
- No Head Left Behind – Multi-Head Alignment Distillation for Transformers (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.