Research map / Architectures and efficiency / Attention and transformers
Focus and Dilution: The Multi-stage Learning Process of Attention
Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu, Tao Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add residual connections to the one-layer attention model (ablation) | one-layer Transformer on Markov data · synthetic Markov data | focus-dilution pattern and embedding norm evolution | No clear effect |
| use multiple attention layers instead of one (ablation) | Transformer on Markov data · synthetic Markov data | focus-dilution and embedding retraction | Mixed |
| use multi-head attention instead of single head (ablation) | Transformer on Markov data · synthetic Markov data | embedding retraction | No clear effect |
| train on real text (WikiText) instead of synthetic Markov data | simplified one-layer Transformer · WikiText | attention focus then dilution on high-frequency tokens | No clear effect |
| train on TinyStories instead of synthetic Markov data | simplified one-layer Transformer · TinyStories | attention focus then dilution on high-frequency tokens | No clear effect |
Closest papers by meaning
- High-Dimensional Learning Dynamics of Attention-Indexed Models (NeurIPS 2026 · similarity 0.90)
- Bayes optimal learning of attention-indexed models (NeurIPS 2025 · similarity 0.89)
- Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence (NeurIPS 2026 · similarity 0.89)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.88)
- Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape (ICML 2024 · similarity 0.88)
- Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
