Research map / Architectures and efficiency / Pruning and efficient tuning
Towards Making the Most of BERT in Neural Machine Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine rate-scheduling, dynamic switch and asymptotic distillation | Transformer base NMT · WMT14 En-De | BLEU: 27.2 -> 30.1 | Helped |
| combine rate-scheduling, dynamic switch and asymptotic distillation | Transformer base NMT · WMT14 En-Fr | BLEU: 41.0 -> 42.3 | Helped |
| combine rate-scheduling, dynamic switch and asymptotic distillation | Transformer base NMT · WMT En-Zh | BLEU: 37.3 -> 38.9 | Helped |
| combine all three strategies vs best single strategy (ablation) | CTNMT · WMT14 En-De | BLEU: 29.7 -> 30.1 | Helped |
| apply BERT distillation to encoder instead of decoder (ablation) | CTNMT AD · WMT14 En-De | BLEU: 26.1 -> 29.2 | Helped |
| use BERT instead of GPT-2 as teacher for encoder (ablation) | CTNMT AD · WMT14 En-De | BLEU: 27.7 -> 29.2 | Helped |
| use asymptotic distillation instead of fine-tuning (ablation) | Transformer NMT · WMT14 En-De subsets | BLEU: 27.8 -> 29.2 | Helped |
| use dynamic switch gate instead of average pooling (ablation) | CTNMT · WMT14 En-De | BLEU: 28.5 -> 29.4 | Helped |
| keep NMT encoder rather than replacing it with frozen BERT features (ablation) | CTNMT · WMT14 En-De | BLEU: 25.2 -> 29.4 | Helped |
| use slanted-triangular rate-scheduled LM learning rate instead of full fine-tuning (ablation) | CTNMT · WMT14 En-De | BLEU: 27.7 -> 29.7 | Helped |
| use slanted-triangular schedule instead of fixed small rate (ablation) | CTNMT · WMT14 En-De | BLEU: 29.0 -> 29.7 | Helped |
| distill from second-to-last instead of last BERT layer (ablation) | CTNMT AD · WMT14 En-De | BLEU: 28.4 -> 29.2 | Helped |
Closest papers by meaning
- Acquiring Knowledge from Pre-Trained Model to Neural Machine Translation (AAAI 2020 · similarity 0.93)
- Deep Fusing Pre-trained Models into Neural Machine Translation (AAAI 2022 · similarity 0.90)
- Incorporating BERT into Neural Machine Translation (ICLR 2020 · similarity 0.90)
- Prompting Neural Machine Translation with Translation Memories (AAAI 2023 · similarity 0.89)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.88)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.