Research map / Architectures and efficiency / Attention and transformers
Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add layer-wise coordination (mixed attention, layer alignment, weight sharing) | Transformer small · IWSLT14 De-En | BLEU: 32.86 -> 35.07 | Helped |
| add layer-wise coordination | Transformer big · WMT16 En-Ro | BLEU: 32.70 -> 34.43 | Helped |
| add layer-wise coordination | Transformer big · WMT14 En-De | BLEU: 28.40 -> 29.01 | Helped |
| share encoder/decoder weights (doubling depth at equal params) (ablation) | layer-wise coordination model · IWSLT14 De-En | BLEU: 33.96 -> 35.07 | Helped |
| use mixed attention instead of separate self/source-target attention (ablation) | layer-wise coordination model · IWSLT14 De-En | BLEU: 33.77 -> 35.07 | Helped |
| add source/target language embedding (ablation) | layer-wise coordination model · IWSLT14 De-En | BLEU: 32.80 -> 35.07 | Helped |
| add resettable position embedding (ablation) | layer-wise coordination model · IWSLT14 De-En | BLEU: 18.46 -> 35.07 | Helped |
| increase depth 14 -> 18 layers (ablation) | layer-wise coordination model · IWSLT14 De-En | BLEU: 35.07 -> 35.31 | Helped |
| increase depth 18 -> 22 layers (ablation) | layer-wise coordination model · IWSLT14 De-En | BLEU: 35.31 -> 35.05 | Hurt |
| increase baseline Transformer depth 6 -> 10 layers | Transformer small · IWSLT14 De-En | BLEU: 32.86 -> 32.67 | No clear effect |
| mixed attention vs context gate at matched params | Transformer without weight sharing · IWSLT14 De-En | BLEU: 33.02 -> 33.96 | Helped |
Closest papers by meaning
- Deep Fusing Pre-trained Models into Neural Machine Translation (AAAI 2022 · similarity 0.90)
- Multi-Channel Encoder for Neural Machine Translation (AAAI 2018 · similarity 0.89)
- Neural Machine Translation with Joint Representation (AAAI 2020 · similarity 0.89)
- Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement (AAAI 2019 · similarity 0.89)
- Tied Transformers: Neural Machine Translation with Shared Encoder and Decoder (AAAI 2019 · similarity 0.88)
- Recurrent Stacking of Layers for Compact Neural Machine Translation Models (AAAI 2019 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.