Research map / Architectures and efficiency / Attention and transformers
Monotonic Multihead Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Monotonic Multihead Attention instead of MILk | Simultaneous machine translation · IWSLT15 En-Vi | BLEU at a given latency | Helped |
| Use Monotonic Multihead Attention instead of MILk | Simultaneous machine translation · WMT15 De-En | BLEU at a given latency | Helped |
| Increase the variance-loss weight (ablation) | MMA-H · IWSLT15 En-Vi and WMT15 De-En test sets | average attention span | Helped |
| Increase the number of decoder attention heads (ablation) | Offline Transformer and MMA-H · WMT13 validation | BLEU | Helped |
| Increase the number of decoder attention heads (ablation) | Offline Transformer and MMA-H · WMT13 validation | DAL | Hurt |
| Increase the number of decoder attention layers (ablation) | Offline Transformer and MMA-H · WMT13 validation | BLEU | Helped |
| Increase the number of decoder attention layers (ablation) | Offline Transformer and MMA-H · WMT13 validation | DAL | Hurt |
Closest papers by meaning
- Are Sixteen Heads Really Better than One? (NeurIPS 2019 · similarity 0.88)
- Online and Linear-Time Attention by Enforcing Monotonic Alignments (ICML 2017 · similarity 0.88)
- Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling (NeurIPS 2021 · similarity 0.88)
- Translation Heads: Disentangling meaning from language in LLM-based machine translation (ICML 2026 · similarity 0.87)
- Generating Diverse Translation by Manipulating Multi-Head Attention (AAAI 2020 · similarity 0.87)
- MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.