Research map / Architectures and efficiency / Attention and transformers
On the Convergence of Encoder-only Shallow Transformers
Yongtao Wu, Fanghui Liu, Grigorios Chrysos, Volkan Cevher
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use τ₀ = dₘ⁻¹/² rather than τ₀ = dₘ⁻¹ | Shallow Transformer · MNIST, especially at large width | convergence rate: d_m = 1024 and 16384 | Improved |
| Use τ₀ = dₘ⁻¹/² rather than τ₀ = dₘ⁻¹ | Shallow Transformer · Small-width convergence | convergence rate | No clear effect |
| Use LeCun or He initialization instead of NTK initialization | Shallow Transformer · Synthetic regression | convergence rate | Improved |
| Represent vector inputs along the embedding dimension (dₛ = 1, d = 100) (ablation) | Shallow Transformer · Synthetic vector-input regression | training convergence | Improved |
| Represent vector inputs along the sequence dimension (dₛ = 100, d = 1) (ablation) | Shallow Transformer · Synthetic vector-input regression | training convergence | Hurt |
| Increase width from the sub-linear regime to the tested linear-or-larger widths (ablation) | Shallow Transformer with LeCun initialization and τ₀ = dₘ⁻¹/² · Synthetic regression | training loss convergence: d_m= \{10, 100,1000, 4000\} | Improved |
| Use LeCun or He initialization instead of NTK initialization | Shallow Transformer · Global convergence analysis | convergence rate | Improved |
Closest papers by meaning
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.91)
- Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (ICLR 2024 · similarity 0.89)
- Two Heads are Better than One: Simulating Large Transformers with Small Ones (NeurIPS 2025 · similarity 0.87)
- A unified framework for establishing the universal approximation of transformer-type architectures (NeurIPS 2025 · similarity 0.87)
- Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity (NeurIPS 2026 · similarity 0.87)
- Subcritical Signal Propagation at Initialization in Normalization-Free Transformers (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
