Research map / Architectures and efficiency / Attention and transformers
Recurrent Stacking of Layers for Compact Neural Machine Translation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Share layer parameters across 6 recurrent stack steps instead of using separate layer parameters | Transformer NMT · WMT Tr-En | BLEU: 18.36 -> 17.31 | Hurt |
| Share layer parameters across 6 recurrent stack steps instead of using separate layer parameters | Transformer NMT · WMT En-Tr | BLEU: 16.29 -> 15.77 | Hurt |
| Share layer parameters across 6 recurrent stack steps instead of using separate layer parameters | Transformer NMT · ALT Ja-En | BLEU: 8.47 -> 8.26 | Hurt |
| Share layer parameters across 6 recurrent stack steps instead of using separate layer parameters | Transformer NMT · KFTT Ja-En | BLEU: 27.19 -> 26.51 | Hurt |
| Share layer parameters across 6 recurrent stack steps instead of using separate layer parameters | Transformer NMT · GCP Ja-En | BLEU: 24.67 -> 24.36 | Hurt |
| Share layer parameters across 6 recurrent stack steps instead of using separate layer parameters | Transformer NMT · GCP En-Ja | BLEU: 26.22 -> 25.84 | Hurt |
| Increase recurrent stacking from 1 to 6 steps (ablation) | RSNMT · WMT Tr-En, WMT En-Tr, KFTT Ja-En, GCP Ja-En and GCP En-Ja | BLEU: 13.75 -> 17.31; 11.94 -> 15.77; 21.64 -> 26.51; 21.95 -> 24.36; 23.89 -> 25.84 | Helped |
| Increase recurrent stacking from 1 to 6 steps (ablation) | RSNMT · ALT Ja-En | BLEU: 7.59 -> 8.26 | Helped |
| Add back-translated data (ablation) | RSNMT 6 · GCP Ja-En | BLEU: 24.36 -> 25.82 | Helped |
| Add back-translated data (ablation) | RSNMT 6 · GCP En-Ja | BLEU: 25.84 -> 27.55 | Helped |
| Decode a model trained with 6 recurrent steps using only 1 step (ablation) | RSNMT 6 · GCP En-Ja | BLEU: 25.84 -> 2.56 | Hurt |
| Decode a model trained with 6 recurrent steps using 5 steps (ablation) | RSNMT 6 · GCP En-Ja | BLEU: 25.84 -> 25.81 | No clear effect |
Closest papers by meaning
- Quasi-Recurrent Neural Networks (ICLR 2017 · similarity 0.89)
- Universal Transformers (ICLR 2019 · similarity 0.89)
- Memory Architectures in Recurrent Neural Network Language Models (ICLR 2018 · similarity 0.88)
- Language Modeling with Gated Convolutional Networks (ICML 2017 · similarity 0.88)
- Unsupervised Neural Machine Translation (ICLR 2018 · similarity 0.88)
- Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation (NeurIPS 2018 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.