Research map / Label-efficient and robust learning / Knowledge distillation
Multilingual Neural Machine Translation with Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill the multilingual model from individual teachers | Multi-Baseline · IWSLT, 12 language pairs | BLEU: +2.01 | Helped |
| distill the multilingual model from individual teachers | Multi-Baseline · WMT, 6 language pairs | BLEU: +1.55 | Helped |
| distill the multilingual model from individual teachers | Multilingual baseline · Ted Talk, 44 languages→English | BLEU: 1.82 | Helped |
| use selective distillation instead of distilling all the time (ablation) | Multilingual model · Ted Talk, 16 reported languages | BLEU: +1.11; +2.99; +2.10; +0.63; +1.62; +2.14; +1.29; +4.23; +4.88; +0.55; +1.15; +1.58; +2.17; -0.75; -0.54; -0.96 | Mixed |
| increase Top-K from 1 to 8 (ablation) | Multilingual distillation · IWSLT De→En validation | BLEU: 33.45 -> 34.76 | Helped |
| increase Top-K beyond 8, including using the full distribution (ablation) | Multilingual distillation · IWSLT De→En validation | BLEU: 34.76; 34.66; 34.68; 34.54; 34.47; 34.49 | No clear effect |
| add back distillation from the multilingual model to individual models (ablation) | Individual models · IWSLT, 9 languages reported | BLEU: +0.20; +1.40; +0.64; +0.92; +0.36; +0.32; +1.18; +0.84; +1.11 | Helped |
| use sequence-level instead of word-level distillation (ablation) | Multilingual model · IWSLT En→12 languages | BLEU: 1.01; 1.68; 0.87; 1.50; 0.53; 1.10; 1.36; 1.04; 0.10; 1.73; 0.61; 0.16 | Hurt |
Closest papers by meaning
- Understanding Knowledge Distillation in Non-autoregressive Machine Translation (ICLR 2020 · similarity 0.90)
- Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation (AAAI 2023 · similarity 0.90)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.87)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.87)
- Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models (AAAI 2025 · similarity 0.86)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.