Research map / Architectures and efficiency / Attention and transformers
Generating Diverse Translation by Manipulating Multi-Head Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use attention-head sampling with K=3 instead of ordinary beam decoding | Transformer base · NIST Zh-En MT03–MT06 average | DEQ: 10.70 | Helped |
| Use attention-head sampling with K=4 instead of ordinary beam decoding | Transformer base · WMT14 En-De | pair-wise BLEU (pwb): 80.41 -> 62.04 | Helped |
| Use attention-head sampling with K=4 instead of ordinary beam decoding | Transformer base · WMT16 En-Ro | pair-wise BLEU (pwb): 81.29 -> 71.12 | Helped |
| Increase K from 3 to 4 (ablation) | Attention-head sampling · NIST Zh-En MT03–MT06 average | pair-wise BLEU (pwb): 66.18 -> 51.66 | Helped |
| Increase K from 3 to 4 (ablation) | Attention-head sampling · NIST Zh-En MT03–MT06 average | reference BLEU (rfb): 42.66 -> 39.80 | Hurt |
| Increase K from 3 to 8 (ablation) | Attention-head sampling · NIST Zh-En MT03–MT06 average | pair-wise BLEU (pwb): 66.18 -> 37.82 | Helped |
| Increase K from 3 to 8 (ablation) | Attention-head sampling · NIST Zh-En MT03–MT06 average | reference BLEU (rfb): 42.66 -> 35.20 | Hurt |
| Use attention-head sampling for self back-translation rather than unrestricted sampling | NIST Zh-En back-translation · NIST Zh-En MT03–MT06 average | BLEU: 45.91 -> 46.14 | Helped |
| Use attention-head sampling for back-translation rather than unrestricted sampling | NIST Zh-En with additional monolingual data · NIST Zh-En | BLEU: 13.38 -> 14.03 | Helped |
| Apply attention-head sampling with K=3 | Seq2Seq conversation response generation · STC | Div: 52.94 -> 39.82 | Helped |
| Increase K from 3 to 8 (ablation) | Attention-head sampling for STC response generation · STC | BLEU: 13.48 -> 12.73 | Hurt |
| Use 2 decoder layers instead of 6 (ablation) | Transformer · NIST Zh-En | Speed: 1 X -> 2.34 X | Helped |
Closest papers by meaning
- Are Sixteen Heads Really Better than One? (NeurIPS 2019 · similarity 0.87)
- Monotonic Multihead Attention (ICLR 2020 · similarity 0.87)
- Translation Heads: Unveiling Attention's Role in LLM Multilingual Translation (ICLR 2026 · similarity 0.86)
- Exploring the Translation Mechanism of Large Language Models (NeurIPS 2025 · similarity 0.85)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.85)
- KV Shifting Attention Enhances Language Modeling (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.