Research map / Architectures and efficiency / Attention and transformers
Attention is All you Need
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adding the Transformer base model (no recurrence or convolution) instead of prior recurrent/conv models | EN-DE translation · WMT 2014 EN-DE newstest2014 | BLEU: 27.3 vs 26.36 for ConvS2S Ensemble | Helped |
| Scaling to the big Transformer (d_model 1024, 16 heads) | EN-DE translation · WMT 2014 EN-DE newstest2014 | BLEU: 28.4 vs 27.3 base | Helped |
| Using big Transformer instead of prior single models | EN-FR translation · WMT 2014 EN-FR newstest2014 | BLEU: 41.0 vs 40.46 for ConvS2S single | Helped |
| Single-head attention instead of 8 heads (ablation) | Transformer base · WMT 2014 EN-DE newstest2013 (dev) | BLEU: 24.9 vs 25.8 | Hurt |
| Removing dropout (P_drop=0.0) (ablation) | Transformer base · WMT 2014 EN-DE newstest2013 (dev) | BLEU: 24.6 vs 25.8 | Hurt |
| Reducing attention key size d_k to 16 (ablation) | Transformer base · WMT 2014 EN-DE newstest2013 (dev) | BLEU: 25.1 vs 25.8 | Hurt |
| Replacing sinusoidal positional encoding with learned positional embeddings (ablation) | Transformer base · WMT 2014 EN-DE newstest2013 (dev) | BLEU: 25.7 vs 25.8 | No clear effect |
| Increasing number of heads beyond 8 (32 heads) (ablation) | Transformer base · WMT 2014 EN-DE newstest2013 (dev) | BLEU: 25.4 vs 25.8 | Hurt |
| Larger model (d_ff 4096, bigger capacity) (ablation) | Transformer base · WMT 2014 EN-DE newstest2013 (dev) | BLEU: 26.2 vs 25.8 | Helped |
Closest papers by meaning
- Neural Machine Translation with Joint Representation (AAAI 2020 · similarity 0.89)
- Posterior Attention Models for Sequence to Sequence Learning (ICLR 2019 · similarity 0.89)
- Asynchronous Bidirectional Decoding for Neural Machine Translation (AAAI 2018 · similarity 0.88)
- Syntax-Directed Attention for Neural Machine Translation (AAAI 2018 · similarity 0.88)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.86)
- Exploring the Translation Mechanism of Large Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.