Research map / Architectures and efficiency / Attention and transformers
Why "Classic" Transformers Are Shallow and A Depth-Enabling Technique
Yueyao Yu, Yin Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add de-escalation step (I - tau*Pi_1) with tau=1 to each post-norm block | 80-layer post-norm ViT · CIFAR10 (ViT depth 80) | train/test loss | Improved |
| add de-escalation step to the input of each FFN block | 60-layer post-norm Transformer-XL · WikiText-103 (depth 60) | training loss | Improved |
| de-escalated post-norm versus pre-norm | ViT depth 80 and Transformer-XL depth 60 · CIFAR10 and WikiText-103 | training loss | No clear effect |
| increase de-escalation strength tau from 0 up to 0.4 (ablation) | Vision Transformer, softmax attention, 40 layers · token diversity dynamics | token diversity | Improved |
| place the de-escalation operator at other locations in the block (ablation) | Vision Transformer, softmax attention · token diversity dynamics | token diversity | No clear effect |
| use pre-norm instead of post-norm architecture | Qwen2 and Llama2 (40 layers) · token similarity across layers | token similarity: remains below 0.8 after 40 layers | Improved |
| apply triangular causal mask to attention (ablation) | Vision Transformer · token similarity escalation | token similarity | No clear effect |
| use alternative attention formulas instead of softmax (ablation) | Transformer with random weights · token similarity escalation | token similarity | No clear effect |
Closest papers by meaning
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.89)
- A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers (NeurIPS 2025 · similarity 0.89)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.89)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (ICLR 2025 · similarity 0.88)
- What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
