Research map / Architectures and efficiency / Attention and transformers
Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer
Yuandong Tian, Yiping Wang, Beidi Chen, Simon Shaolei Du
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Attend to distinct contextual tokens rather than common tokens | 1-layer Transformer self-attention · sequence-class setting | attention weight on distinct tokens | Improved |
| Suppress a common contextual token | 1-layer Transformer self-attention · sequence-class setting with one common token | attention weight on common token | Hurt |
| Increase the decoder learning rate eta_Y while holding eta_Z fixed (ablation) | 1-layer Transformer · Syn-Small | attention sparsity | Hurt |
| Increase the self-attention learning rate eta_Z (ablation) | 1-layer Transformer · Syn-Medium | distinct-token attention entropy | Improved |
| Increase eta_Y while holding eta_Z fixed (ablation) | 1-layer Transformer · Syn-Medium | distinct-token attention entropy: eta_Y/eta_Z\approx 2 | Mixed |
| Use Adam instead of SGD (ablation) | 1-layer Transformer · Syn-Small | distinct-token selection | Mixed |
| Use a larger decoder learning rate eta_Y (ablation) | 1-layer Transformer · training dynamics | phase-transition time: t_0 \approx 2K\ln M/\eta_Y | Improved |
| Scale up both learning rates eta_Y and eta_Z (ablation) | 1-layer Transformer · training dynamics | attention sparsity | Improved |
| Decrease eta_Y while holding eta_Z fixed (ablation) | 1-layer Transformer · after phase transition | attention sparsity | Improved |
| Train on WikiText with SGD | 1-layer and multi-layer Transformers · WikiText | first-layer attention sparsity | Improved |
Closest papers by meaning
- Non-asymptotic Convergence of Training Transformers for Next-token Prediction (NeurIPS 2024 · similarity 0.90)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.90)
- Looking Beyond the Top-1: Transformers Determine Top Tokens in Order (ICML 2025 · similarity 0.89)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.89)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.89)
- JoMA: Demystifying Multilayer Transformers via Joint Dynamics of MLP and Attention (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
