Research map / Label-efficient and robust learning / Recognition and long-tail
Incorporating BERT into Neural Machine Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse fixed BERT representations through attention in every encoder and decoder layer | Transformer NMT · IWSLT'14 En→De | BLEU: 28.57 -> 30.45 | Helped |
| Use document context from the preceding sentence with BERT-fused translation | BERT-fused model · IWSLT'14 document-level De→En | BLEU: 36.11 -> 36.69 | Helped |
| Add BERT-fused modeling after back-translation training | Transformer + back translation · WMT'16 Ro→En | BLEU: 37.73 -> 39.10 | Helped |
| Warm-start the NMT encoder and decoder rather than randomly initialize them (ablation) | BERT-fused model · IWSLT'14 En→De | BLEU: 27.03 -> 30.45 | Helped |
| Keep BERT fixed rather than jointly tune it with the NMT modules (ablation) | BERT-fused model · IWSLT'14 En→De | BLEU: 28.87 -> 30.45 | Helped |
| Use attention to fuse BERT features rather than feed them directly into every layer (ablation) | BERT-fused model · IWSLT'14 En→De | BLEU: 29.61 -> 30.45 | Helped |
| Use pretrained BERT features instead of random vectors (ablation) | BERT-fused model · IWSLT'14 En→De | BLEU: 28.91 -> 30.45 | Helped |
| Use BERT instead of another NMT Transformer encoder as the external feature source (ablation) | BERT-fused model · IWSLT'14 En→De | BLEU: 28.99 -> 30.45 | Helped |
| Add BERT-encoder attention (ablation) | BERT-fused model · IWSLT'14 En→De | BLEU: 29.87 -> 30.45 | Helped |
| Add BERT-decoder attention (ablation) | BERT-fused model · IWSLT'14 En→De | BLEU: 29.90 -> 30.45 | Helped |
| Apply the three attention operations sequentially rather than fuse them as proposed (ablation) | BERT-fused model · IWSLT'14 En→De | BLEU: 30.45 -> 29.35 | Hurt |
| Increase the Transformer encoder depth to 12 layers (ablation) | Standard Transformer · IWSLT'14 En→De | BLEU: 28.57 -> 29.27 | Helped |
Closest papers by meaning
- Towards Making the Most of BERT in Neural Machine Translation (AAAI 2020 · similarity 0.90)
- Acquiring Knowledge from Pre-Trained Model to Neural Machine Translation (AAAI 2020 · similarity 0.88)
- From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables (AAAI 2022 · similarity 0.88)
- Incorporating BERT into Parallel Sequence Decoding with Adapters (NeurIPS 2020 · similarity 0.88)
- Semantics-Aware BERT for Language Understanding (AAAI 2020 · similarity 0.88)
- Neural Machine Translation with Joint Representation (AAAI 2020 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.