Research map / Architectures and efficiency / Attention and transformers
Limits to Depth Efficiencies of Self-Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Deepen a self-attention network at fixed parameter budget when width is large enough (L below log3 of width) | Transformer encoder · language modeling loss | loss | Helped |
| Deepen a small-width self-attention network at fixed parameter budget | Transformer encoder · language modeling loss | loss | Hurt |
| Reorder sublayers to run all self-attention before feed-forward (Press et al.) | BERT · WikiText-103 test | perplexity: 18.63 -> 18.82 | No clear effect |
Closest papers by meaning
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.86)
- Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.86)
- Do Language Models Use Their Depth Efficiently? (NeurIPS 2025 · similarity 0.86)
- The emergence of sparse attention: impact of data distribution and benefits of repetition (NeurIPS 2025 · similarity 0.86)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.86)
- A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalization (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.