Research map / Architectures and efficiency / Attention and transformers
Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
Bobby He, James Martens, Guodong Zhang, Aleksandar Botev, Andrew Brock, Samuel L Smith, Yee Whye Teh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use E-SPA instead of U-SPA for a 36-block skipless transformer | Skipless transformer · WikiText-103 | training loss: $3.10_{\pm.01}$ -> $\textbf{2.86}_{\pm .04}$ | Improved |
| Use E-SPA instead of Value-SkipInit for a 36-block skipless transformer | Skipless transformer · WikiText-103 | training loss: $3.18_{\pm .01}$ -> $\textbf{2.86}_{\pm .04}$ | Improved |
| Remove skip connections from the standard transformer | Standard transformer · WikiText-103 | training loss: training loss plateauing around 7.5 | Hurt |
| Use sigmoid instead of GeLU in E-SPA (ablation) | 36-block skipless transformer · WikiText-103 | training loss: $\textbf{2.86}_{\pm .04}$ -> $7.42_{\pm.01}$ | Hurt |
| Remove layer normalisation from E-SPA with DKS-GeLU (ablation) | 36-block skipless transformer · WikiText-103 | training loss: $2.99_{\pm .01}$ -> $\textbf{2.79}_{\pm .02}$ | Improved |
| Remove layer normalisation from E-SPA with GeLU (ablation) | 36-block skipless transformer · WikiText-103 | training loss: $\textbf{2.86}_{\pm .04}$ -> $2.93_{\pm .02}$ | Hurt |
| Use TAT-Leaky ReLU instead of GeLU in E-SPA without LN (ablation) | 36-block skipless transformer · WikiText-103 | training loss: $2.89_{\pm.02}$ -> $2.93_{\pm .02}$ | Hurt |
| Use E-SPA instead of U-SPA with normalised skips and layer normalisation | Transformer with normalised skips · C4 | validation perplexity: 27.6 -> 24.0 | Improved |
| Use E-SPA with normalised skips instead of Default Pre-LN | Transformer with layer normalisation · C4 | validation perplexity: 24.7 -> 24.0 | Improved |
| Train vanilla E-SPA for more steps, from 100K to 1000K (ablation) | 36-block transformer · WikiText-103 | training loss: 3.12 -> 2.20 | Improved |
| Increase vanilla E-SPA depth from 36 to 72 blocks at 1000K steps (ablation) | Vanilla E-SPA · WikiText-103 | training loss: 2.20 -> 2.11 | Improved |
| Increase vanilla E-SPA depth from 36 to 108 blocks at 100K steps (ablation) | Vanilla E-SPA · WikiText-103 | training loss: 3.12 -> 3.42 | Hurt |
Closest papers by meaning
- Transformers without Normalization (CVPR 2025 · similarity 0.89)
- Subcritical Signal Propagation at Initialization in Normalization-Free Transformers (NeurIPS 2026 · similarity 0.88)
- Simplifying Transformer Blocks (ICLR 2024 · similarity 0.88)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.88)
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.87)
- Attention-Only Transformers via Unrolled Subspace Denoising (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
