Research map / Architectures and efficiency / Attention and transformers
Tied Transformers: Neural Machine Translation with Shared Encoder and Decoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| tie encoder and decoder weights | Transformer small · IWSLT14 De-En | BLEU: 33.27 -> 35.10 | Helped |
| tie encoder and decoder weights | Transformer small · IWSLT14 En-Ro | BLEU: 23.80 -> 24.31 | Helped |
| tie encoder and decoder weights on dissimilar languages | Transformer small · IWSLT14 En-Ru | BLEU: 14.59 -> 14.90 | Helped |
| tie encoder and decoder weights (vs cross-task sharing) | Transformer small · IWSLT14 De-En | BLEU: 34.33 -> 35.10 | Helped |
| use 6-block tied transformer instead of standard 6-block | Transformer big · WMT14 En-De | BLEU: 28.40 -> 28.35 | No clear effect |
| increase tied transformer depth from 6 to 12 blocks (ablation) | Tied Transformer · WMT14 En-De | BLEU: 28.35 -> 28.98 | Helped |
| add 8M back-translated monolingual data | 8-block Tied Transformer · WMT14 En-De | BLEU: 28.88 -> 29.89 | Helped |
| increase tied transformer from 6 to 8 blocks (ablation) | Tied Transformer · WMT16 En-Ro | BLEU: 34.67 -> 34.55 | Hurt |
| tie encoder and decoder in LSTM, hidden 512 | LSTM NMT · IWSLT14 De-En | BLEU: 29.41 -> 29.57 | No clear effect |
| tie encoder and decoder in LSTM, hidden 1024 | LSTM NMT · IWSLT14 De-En | BLEU: 29.41 -> 30.61 | Helped |
| hidden dimension 128 for tied model (ablation) | Tied Transformer · IWSLT14 De-En | BLEU: 0.82 BLEU lower than baseline | Hurt |
| increase hidden dimension from 256 to 384 in tied model (ablation) | Tied Transformer · IWSLT14 De-En | BLEU: 35.10 -> 35.52 | Helped |
Closest papers by meaning
- Unsupervised Neural Machine Translation (ICLR 2018 · similarity 0.89)
- From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables (AAAI 2022 · similarity 0.89)
- Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation (NeurIPS 2018 · similarity 0.88)
- Neural Machine Translation with Joint Representation (AAAI 2020 · similarity 0.88)
- Deep Transformers with Latent Depth (NeurIPS 2020 · similarity 0.88)
- Neural Machine Translation with Soft Prototype (NeurIPS 2019 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.