Research map / Architectures and efficiency / RNNs and normalization
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the 6-6 autoregressive layer allocation with a 12-1 deep-encoder, shallow-decoder allocation (ablation) | Autoregressive Transformer · WMT14 EN→DE and WMT16 EN→RO | BLEU: 28.3 vs. 27.4 (EN→DE); 33.8 vs. 34.6 (EN→RO) | Mixed |
| Replace the 6-6 autoregressive layer allocation with a 12-1 deep-encoder, shallow-decoder allocation (ablation) | Autoregressive Transformer · WMT17 EN→ZH, WMT17 ZH→EN, WMT14 EN→FR | S1 speedup: 1.0× -> 2.7×; 1.0× -> 2.9×; 1.0× -> 2.8× | Helped |
| Replace the 6-6 autoregressive layer allocation with a 12-1 deep-encoder, shallow-decoder allocation (ablation) | Autoregressive Transformer · WMT17 EN→ZH, WMT17 ZH→EN, WMT14 EN→FR | Smax speedup: 1.0× -> 1.7×; 1.0× -> 1.8×; 1.0× -> 1.9× | Helped |
| Increase autoregressive encoder depth from 6 to 12 while keeping a 1-layer decoder (ablation) | Autoregressive Transformer · WMT14 EN→DE test | BLEU: 27.4 -> 28.3 | Helped |
| Use a 1-layer rather than 6-layer autoregressive decoder with a 6-layer encoder (ablation) | Autoregressive Transformer · WMT14 EN→DE test | BLEU: 28.3 -> 27.4 | Hurt |
| Use a 12-1 rather than 6-6 layer allocation in DisCo (ablation) | DisCo · WMT14 EN→DE test | BLEU: 27.4 -> 26.8 | Hurt |
| Use a 12-1 rather than 6-6 layer allocation in CMLM with T=10 (ablation) | CMLM · WMT14 EN→DE test | BLEU: 27.4 -> 26.3 | Hurt |
| Apply sequence-level knowledge distillation (ablation) | AR and non-autoregressive translation models · WMT14 EN→DE test | BLEU: AR Deep-Shallow: 26.9 -> 28.3; DisCo: 24.8 -> 27.4 | Helped |
| Reduce NAR length beam size from 5 to 1 (ablation) | NAR 6-6 models · WMT14 EN→DE test | BLEU: 0.5 BLEU points | Hurt |
| Remove the decoder feed-forward module (ablation) | Autoregressive deep-shallow model · Translation inference | S1 speed: 10% | Helped |
| Use a 12-1 rather than 6-6 layer allocation in NAR models (ablation) | CMLM and DisCo · WMT14 EN→DE test | BLEU: 27.35 vs. 26.75 | Hurt |
Closest papers by meaning
- Non-autoregressive Translation with Layer-Wise Prediction and Deep Supervision (AAAI 2022 · similarity 0.92)
- Non-Autoregressive Neural Machine Translation (ICLR 2018 · similarity 0.92)
- Improving Non-Autoregressive Translation Models Without Distillation (ICLR 2022 · similarity 0.92)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.89)
- When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models (ICML 2024 · similarity 0.89)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.