AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Why "Classic" Transformers Are Shallow and A Depth-Enabling Technique

Yueyao Yu, Yin Zhang

JMLR 2026 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add de-escalation step (I - tau*Pi_1) with tau=1 to each post-norm block80-layer post-norm ViT · CIFAR10 (ViT depth 80)train/test lossImproved
add de-escalation step to the input of each FFN block60-layer post-norm Transformer-XL · WikiText-103 (depth 60)training lossImproved
de-escalated post-norm versus pre-normViT depth 80 and Transformer-XL depth 60 · CIFAR10 and WikiText-103training lossNo clear effect
increase de-escalation strength tau from 0 up to 0.4 (ablation)Vision Transformer, softmax attention, 40 layers · token diversity dynamicstoken diversityImproved
place the de-escalation operator at other locations in the block (ablation)Vision Transformer, softmax attention · token diversity dynamicstoken diversityNo clear effect
use pre-norm instead of post-norm architectureQwen2 and Llama2 (40 layers) · token similarity across layerstoken similarity: remains below 0.8 after 40 layersImproved
apply triangular causal mask to attention (ablation)Vision Transformer · token similarity escalationtoken similarityNo clear effect
use alternative attention formulas instead of softmax (ablation)Transformer with random weights · token similarity escalationtoken similarityNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy