Research map / Architectures and efficiency / Attention and transformers
Attention is not all you need: pure attention loses rank doubly exponentially with depth
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove skip connections and MLPs from a transformer (pure self-attention network) (ablation) | Pure attention SAN (depth L) · Real architectures BERT, Albert, XLNet residual analysis | Relative residual ||res(SAN_l(X))||/||SAN_l(X)|| across depth | Hurt |
| Add skip connections to self-attention network (ablation) | Self-attention network (SAN) · Real architectures BERT, Albert, XLNet residual analysis | Relative residual across depth (rank collapse) | Helped |
| Add MLP blocks to self-attention network (ablation) | Self-attention network (SAN) · Toy 2D circular trajectory task | Convergence of predicted trajectories at inference (rank collapse) | Helped |
| Add MLP blocks to self-attention network (ablation) | Self-attention network (SAN) · Real architectures BERT, Albert, XLNet residual analysis | Relative residual across depth | No clear effect |
| Increase weight norm beta (hidden dimension 32 to 128) in single-layer recurrent transformer without skip/MLP (ablation) | Single-layer transformer applied recurrently · Toy 2D circular trajectory task | Speed of rank collapse at inference | Helped |
| Add layer normalization (ablation) | Self-attention network · Theoretical analysis | Residual convergence bound | No clear effect |
| Use only short paths (length 0 to 2) versus long paths (length up to 6) in trained attention networks (ablation) | Trained attention networks on memorization, sorting, convex hull · Sequence memorization, sorting, convex hull | Per-token prediction accuracy of path subsets | Mixed |
Closest papers by meaning
- Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.91)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.89)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.88)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.88)
- Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few (NeurIPS 2025 · similarity 0.88)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.