Research map / Architectures and efficiency / Attention and transformers
Improving Transformer Optimization Through Better Initialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove warmup and layer normalization, using T-Fixup initialization | Transformer small · IWSLT'14 De-En | test BLEU: 34.2 -> 35.5 | Helped |
| Remove warmup and layer normalization, using T-Fixup initialization | Transformer base · WMT'17 En-De | test BLEU: 27.3 -> 29.1 | Helped |
| Pre-LN (moving layer norm off main branch) | Transformer big · WMT'17 En-De | test BLEU: 29.3 -> 28.7 | Hurt |
| Increase depth from 6 to 30 layers with T-Fixup (no warmup/LN) (ablation) | Transformer base · WMT'17 En-De | test BLEU: 29.1 -> 29.7 | Helped |
| Increase depth from 12 to 30 layers with T-Fixup (ablation) | Transformer base · WMT'14 En-De | test BLEU: 29.3 -> 30.1 | Helped |
| Train 200-layer encoder-decoder with T-Fixup (64-128-2 config) (ablation) | Transformer small-reduced · IWSLT'14 De-En | validation loss / test BLEU: 200 layers: 27.65 BLEU, trains without divergence | No clear effect |
| Scale batch size 2x-8x without warmup under T-Fixup (ablation) | Transformer base · WMT'17 En-De | test BLEU: BLEU roughly stable across 1x-8x batch size | No clear effect |
| Remove layer normalization from baseline without T-Fixup init (ablation) | Transformer baseline · IWSLT'14 De-En | validation loss: training diverges even with warmup | Hurt |
| Replace Adam with SGD in no-warmup, LN setting (ablation) | Transformer small · IWSLT'14 De-En | validation loss: gets stuck initially then converges, but much slower | Mixed |
Closest papers by meaning
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.90)
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.89)
- What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis (ICLR 2025 · similarity 0.89)
- Simplifying Transformer Blocks (ICLR 2024 · similarity 0.88)
- SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (ICML 2024 · similarity 0.88)
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.