Research map / Architectures and efficiency / Attention and transformers
Learning Multiscale Transformer Models for Sequence Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add linguistic word-boundary and phrase-level interactions | Transformer on WMT'14 En-De · WMT'14 En-De newstest2014, Base and Big | BLEU: 27.63 -> 28.51; 29.31 -> 29.75 | Helped |
| Add relative positional representations | UMST on WMT'14 En-De · WMT'14 En-De newstest2014, Base and Big | BLEU: 28.51 -> 28.90; 29.75 -> 30.15 | Helped |
| Add linguistic multiscale interactions | Transformer on WMT'16 En-Ro · WMT'16 En-Ro newstest-2016 | BLEU: 34.21 -> 34.81 | Helped |
| Add linguistic multiscale interactions | Transformer on CNN-DailyMail · CNN-DailyMail | ROUGE-1: 40.55 -> 41.82 | Helped |
| Remove class embedding (ablation) | UMST on WMT'14 En-De · WMT'14 En-De newstest2014, depth 6-6 and 12-6 | BLEU: 28.51 -> 28.39; 29.49 -> 28.99 | Hurt |
| Remove intra-group interactions (ablation) | UMST on WMT'14 En-De · WMT'14 En-De newstest2014, depth 6-6 | BLEU: 28.51 -> 27.87 | Hurt |
| Remove inter-group interactions (ablation) | UMST on WMT'14 En-De · WMT'14 En-De newstest2014, depth 6-6 and 12-6 | BLEU: 28.51 -> 28.06; 29.49 -> 29.37 | Hurt |
| Remove inter-group interactions (ablation) | UMST on CNN-DailyMail · CNN-DailyMail | ROUGE-1: 41.82 -> 41.62 | Hurt |
| Replace GCN with average pooling (ablation) | UMST on WMT'14 En-De · WMT'14 En-De newstest2014, depth 6-6 and 12-6 | BLEU: 28.51 -> 27.96; 29.49 -> 28.89 | Hurt |
| Replace GCN with GAT (ablation) | UMST on WMT'14 En-De · WMT'14 En-De newstest2014, depth 6-6 | BLEU: 28.51 -> 28.11 | Hurt |
| Share the word and phrase GCN projections across blocks, within blocks, or share Q and K (ablation) | UMST on WMT'14 En-De · WMT'14 En-De newstest2014 | BLEU: 28.51 -> 28.33; 28.51 -> 28.39; 28.51 -> 28.41 | No clear effect |
| Replace the proposed up-sampling mapping with nearest or linear interpolation (ablation) | UMST on WMT'14 En-De · WMT'14 En-De newstest2014 | BLEU: 28.51 -> 27.96; 28.51 -> 27.61 | Hurt |
Closest papers by meaning
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.87)
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.87)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
- Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers (NeurIPS 2024 · similarity 0.87)
- Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape (ICML 2024 · similarity 0.86)
- Core Context Aware Transformers for Long Context Language Modeling (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.