AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation

Bobby He, James Martens, Guodong Zhang, Aleksandar Botev, Andrew Brock, Samuel L Smith, Yee Whye Teh

ICLR 2023 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 5 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use E-SPA instead of U-SPA for a 36-block skipless transformerSkipless transformer · WikiText-103training loss: $3.10_{\pm.01}$ -> $\textbf{2.86}_{\pm .04}$Improved
Use E-SPA instead of Value-SkipInit for a 36-block skipless transformerSkipless transformer · WikiText-103training loss: $3.18_{\pm .01}$ -> $\textbf{2.86}_{\pm .04}$Improved
Remove skip connections from the standard transformerStandard transformer · WikiText-103training loss: training loss plateauing around 7.5Hurt
Use sigmoid instead of GeLU in E-SPA (ablation)36-block skipless transformer · WikiText-103training loss: $\textbf{2.86}_{\pm .04}$ -> $7.42_{\pm.01}$Hurt
Remove layer normalisation from E-SPA with DKS-GeLU (ablation)36-block skipless transformer · WikiText-103training loss: $2.99_{\pm .01}$ -> $\textbf{2.79}_{\pm .02}$Improved
Remove layer normalisation from E-SPA with GeLU (ablation)36-block skipless transformer · WikiText-103training loss: $\textbf{2.86}_{\pm .04}$ -> $2.93_{\pm .02}$Hurt
Use TAT-Leaky ReLU instead of GeLU in E-SPA without LN (ablation)36-block skipless transformer · WikiText-103training loss: $2.89_{\pm.02}$ -> $2.93_{\pm .02}$Hurt
Use E-SPA instead of U-SPA with normalised skips and layer normalisationTransformer with normalised skips · C4validation perplexity: 27.6 -> 24.0Improved
Use E-SPA with normalised skips instead of Default Pre-LNTransformer with layer normalisation · C4validation perplexity: 24.7 -> 24.0Improved
Train vanilla E-SPA for more steps, from 100K to 1000K (ablation)36-block transformer · WikiText-103training loss: 3.12 -> 2.20Improved
Increase vanilla E-SPA depth from 36 to 72 blocks at 1000K steps (ablation)Vanilla E-SPA · WikiText-103training loss: 2.20 -> 2.11Improved
Increase vanilla E-SPA depth from 36 to 108 blocks at 100K steps (ablation)Vanilla E-SPA · WikiText-103training loss: 3.12 -> 3.42Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy