Research map / Architectures and efficiency / Attention and transformers
Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Aggregate layers with static linear-combination weights | Transformer-Base · WMT14 English⇒German | BLEU: 27.31 -> 27.73; +0.42 | Helped |
| Use position-specific feed-forward weights for layer aggregation | Transformer-Base · WMT14 English⇒German | BLEU: 27.31 -> 28.33; +1.02 | Helped |
| Use iterative dynamic routing for layer aggregation | Transformer-Base · WMT14 English⇒German | BLEU: 27.31 -> 28.22; +0.91 | Helped |
| Use iterative EM routing instead of static aggregation | Transformer-Base · WMT14 English⇒German | BLEU: 27.31 -> 28.81; +1.50 | Helped |
| Replace static aggregation with dynamic aggregation | Transformer-Base · WMT14 English⇒German | BLEU: 27.73 -> 28.33; 27.73 -> 28.22; 27.73 -> 28.81 | Helped |
| Add EM routing to the base Transformer | Transformer-Base · WMT14 English⇒German | BLEU: 27.31 -> 28.81 | Helped |
| Add EM routing to the base Transformer | Transformer-Base · WMT17 Chinese⇒English | BLEU: 24.13 -> 24.81 | Helped |
| Add EM routing to the big Transformer | Transformer-Big · WMT14 English⇒German | BLEU: 28.58 -> 28.97 | Helped |
| Add EM routing to the big Transformer | Transformer-Big · WMT17 Chinese⇒English | BLEU: 24.56 -> 25.00 | Helped |
| Construct each routing input capsule from all layer representations rather than its corresponding layer alone (ablation) | EM Routing · WMT14 English⇒German development set | BLEU: 26.18 -> 26.62 | Helped |
| Apply EM routing to both encoder and decoder rather than either component alone (ablation) | Transformer-Base · WMT14 English⇒German development set | BLEU: 26.33 -> 26.62; 26.34 -> 26.62 | Helped |
| Increase the number of output capsules (ablation) | Transformer-Base + EM Routing · WMT14 English⇒German development set | BLEU | Helped |
Closest papers by meaning
- Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation (NeurIPS 2018 · similarity 0.89)
- Recurrent Stacking of Layers for Compact Neural Machine Translation Models (AAAI 2019 · similarity 0.88)
- Deep Transformers with Latent Depth (NeurIPS 2020 · similarity 0.86)
- Neuron Interaction Based Representation Composition for Neural Machine Translation (AAAI 2020 · similarity 0.86)
- Neural Machine Translation with Joint Representation (AAAI 2020 · similarity 0.85)
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.