Research map / Architectures and efficiency / Attention and transformers
Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
Xuebo Liu, Longyue Wang, Derek F. Wong, Liang Ding, Lidia S. Chao, Zhaopeng Tu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hard fusion of source-surface probability | Transformer Seq2Seq · WMT16 Ro-En | BLEU: 33.8 -> 35.1 | Improved |
| Add hard fusion of source-surface probability | Transformer Seq2Seq · WMT14 En-De | BLEU: 28.9 -> 29.5 | Improved |
| Add hard fusion of source-surface probability | Transformer Seq2Seq · WMT14 En-Fr | BLEU: 43.4 -> 43.9 | Improved |
| Add soft fusion at the logit level | Transformer Seq2Seq · CNN/Daily Mail | ROUGE-L: 37.2 -> 37.9 | Improved |
| Add soft fusion at the logit level | Transformer Seq2Seq · CONLL14 | F0.5: 54.3 -> 56.6 | Improved |
| Use soft rather than hard fusion | SurfaceFusion · Translation benchmarks | BLEU | Mixed |
| Use soft rather than hard fusion | SurfaceFusion · CNN/Daily Mail and CONLL14 | ROUGE-L / F0.5 | Improved |
| Fuse encoder embedding features through fine-grained layer attention | Vanilla Transformer · WMT16 Ro-En | BLEU: 33.80 -> 34.45 | Improved |
| Add fine-grained layer attention | Vanilla Transformer · Machine translation, grammatical error correction, and summarization | Task scores | Improved |
| Add only the encoder embedding layer connection for the uppermost decoder layer (ablation) | Fine-grained layer attention · Seq2Seq tasks | Task performance | No clear effect |
| Mask the encoder embedding layer (ablation) | Fine-grained layer attention · Machine translation, summarization, and grammatical error correction | Model performance | Hurt |
| Mask the encoder embedding layer (ablation) | Fine-grained layer attention · Machine translation, summarization, and grammatical error correction | Output length | Mixed |
Closest papers by meaning
- Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.86)
- EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.86)
- On Affine Homotopy between Language Encoders (NeurIPS 2024 · similarity 0.86)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.85)
- Layer-Wise Representation Fusion for Compositional Generalization (AAAI 2024 · similarity 0.85)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
