Research map / Architectures and efficiency / Attention and transformers
Mega: Moving Average Equipped Gated Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EMA-equipped gated attention in place of standard Transformer attention | Mega · LRA (Avg.) | accuracy: 59.24 -> 88.21 | Helped |
| Chunk attention into fixed-size blocks, retaining EMA (ablation) | Mega · LRA (Avg.) | accuracy: 88.21 -> 85.66 | Hurt |
| Add softmax Mega attention instead of Transformer-base | WMT'16 EN-DE test · WMT'16 EN-DE test | tokenized BLEU: 27.97 -> 29.18 | Helped |
| Use Laplace attention instead of softmax (ablation) | Mega · WMT'16 EN-DE test | tokenized BLEU: 29.18 -> 28.95 | Hurt |
| Use Mega instead of XFM-XL | WikiText-103 · WikiText-103 test | perplexity: 18.30 -> 18.07 | Helped |
| Use Mega instead of DeiT-B | ImageNet-1k · ImageNet-1k val | top-1 accuracy: 81.8 -> 82.3 | Helped |
| Increase Mega-chunk model size from base to big (ablation) | Speech Commands SC10 · Speech Commands SC10 raw | accuracy: 96.92 -> 97.30 | Helped |
| Remove the EMA component (ablation) | Mega · LRA Text and Image | accuracy | Hurt |
| Use single-dimensional EMA (h=1) rather than removing EMA (ablation) | Mega · LRA Text and Image | accuracy | Helped |
| Use softmax rather than relu² attention (ablation) | Mega · LRA Text | accuracy: 90.08 -> 90.43 | Helped |
| Use Laplace rather than softmax attention (ablation) | Mega · LRA Image | accuracy: 89.87 -> 90.43 | Helped |
| Increase context length at inference (ablation) | Mega · WikiText-103 test | perplexity | Helped |
Closest papers by meaning
- Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.88)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.88)
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.88)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.87)
- Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences (ICML 2024 · similarity 0.87)
- Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.