Research map / Architectures and efficiency / Attention and transformers
Pay Attention to MLPs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multiplicative gating instead of linear spatial projection (ablation) | gMLP · MLM validation | perplexity (lower better): 5.14 -> 4.53 | Helped |
| use multiplicative gating instead of additive (ablation) | gMLP · MLM validation | perplexity (lower better): 4.97 -> 4.53 | Helped |
| split Z into two channel halves for gate and bypass (ablation) | gMLP · MLM validation | perplexity (lower better): 4.53 -> 4.35 | Helped |
| use gMLP SGU instead of MLP-Mixer spatial MLP (ablation) | MLM model · MLM validation | perplexity (lower better): 5.34 -> 4.35 | Helped |
| remove self-attention content terms from Transformer (ablation) | BERT-base + rel pos · MLM validation | perplexity (lower better): 4.26 -> 5.64 | Hurt |
| add tiny single-head self-attention (aMLP) (ablation) | gMLP-large · SQuAD v2.0 | F1: 78.3 -> 85.4 | Helped |
| add tiny single-head self-attention (aMLP) (ablation) | gMLP-base · MNLI-m | accuracy (%): 83.7 -> 85.9 | Helped |
| add tiny attention (aMLP) (ablation) | gMLP-base · SST-2 | accuracy (%): 94.2 -> 93.4 | No clear effect |
| replace self-attention with gMLP | BERT-base · MNLI-m | accuracy (%): 85.6 -> 83.7 | Hurt |
| scale gMLP from base to large | gMLP · SQuAD v2.0 | F1: 70.1 -> 78.3 | Helped |
| replace self-attention with gMLP | BERT-base · ImageNet-1K | top-1 accuracy (%): 81.8 -> 81.6 | No clear effect |
| replace self-attention with gMLP | BERT-large · MLM perplexity | perplexity (lower better): 3.35 -> 3.32 | No clear effect |
Closest papers by meaning
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.86)
- Graph Convolutions Enrich the Self-Attention in Transformers! (NeurIPS 2024 · similarity 0.86)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.86)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.86)
- Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024 · similarity 0.86)
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.