Research map / Label-efficient and robust learning / Knowledge distillation
Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use selective rather than standard knowledge distillation | DeepShallow · WMT14 En-De | BLEU: 27.05 -> 27.23 | Helped |
| Use selective rather than standard knowledge distillation | DeepShallow · WMT14 De-En | BLEU: 31.36 -> 31.70 | Helped |
| Use selective rather than standard knowledge distillation | CMLM · WMT16 En-Ro | BLEU: 32.85 -> 33.38 | Helped |
| Use selective rather than standard knowledge distillation | GLAT + CTC · WMT16 En-Ro | BLEU: 32.73 -> 33.34 | Helped |
| Use selective rather than standard knowledge distillation | GLAT + CTC · WMT14 En-De | word repetition ratio: 1.06\textperthousand -> 0.82\textperthousand | Helped |
| Use selective rather than standard knowledge distillation | GLAT + CTC · WMT14 De-En | word repetition ratio: 0.56\textperthousand -> 0.38\textperthousand | Helped |
| Use selective rather than standard knowledge distillation | GLAT + CTC · WMT16 En-Ro | word repetition ratio: 0.80\textperthousand -> 0.64\textperthousand | Helped |
| Distill only 5\% of the training data using selected raw examples (ablation) | GLAT + CTC · WMT14 En-De | BLEU: +2.4 BLEU | Helped |
| Use 80\%-distilled data rather than fully distilled data (ablation) | GLAT + CTC · WMT14 En-De | BLEU | Helped |
| Use a dynamic threshold rather than a fixed threshold (ablation) | GLAT + CTC · WMT14 En-De | BLEU | Helped |
| Initialize the student with teacher parameters trained for 25k updates rather than training from scratch (ablation) | GLAT + CTC · WMT14 En-De | BLEU: 0.47 BLEU | Helped |
| Vary teacher initialization from 25k to 300k updates (ablation) | GLAT + CTC · WMT14 En-De | BLEU | No clear effect |
Closest papers by meaning
- Understanding Knowledge Distillation in Non-autoregressive Machine Translation (ICLR 2020 · similarity 0.95)
- Non-Autoregressive Neural Machine Translation (ICLR 2018 · similarity 0.90)
- Improving Non-Autoregressive Translation Models Without Distillation (ICLR 2022 · similarity 0.90)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.87)
- DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation (NeurIPS 2024 · similarity 0.86)
- AEQA-NAT : Adaptive End-to-end Quantization Alignment Training Framework for Non-autoregressive Machine Translation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.