Research map / Architectures and efficiency / Attention and transformers
Stabilizing Transformer Training by Preventing Attention Entropy Collapse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply σReparam instead of the standard DeiT recipe | ViT-B/16 · ImageNet1k | Top-1 accuracy (%): 81.8 -> 82.2 | Helped |
| Add a learned scalar to spectral normalization | ViT-B/16 · ImageNet1k | Top-1 accuracy (%): 69.81 -> 82.2 | Helped |
| Use σReparam while retaining pre-LN (ablation) | SimCLR ViT-B/16 · ImageNet1k linear probe, 300 epochs | Top-1 accuracy: 73.7 -> 74.5 | Helped |
| Add σReparam to post-LN | 18L-18L Transformer · WMT'17 English-German newstest2017 | Divergent runs: 1/3 -> 0/3 | Helped |
| Add σReparam to post-LN | 50L-50L Transformer · WMT'17 English-German newstest2017 | Divergent runs: 3/3 -> 0/3 | Helped |
| Add σReparam to DeepNorm | 100L-100L Transformer · WMT'17 English-German newstest2017 | Divergent runs: 2/3 -> 0/3 | Helped |
| Add σReparam to DeepNorm | 50L-50L Transformer · WMT'17 English-German newstest2017 | Test BLEU: 29.2$_{0.1}$ -> 28.3$_{0.3}$ | Hurt |
| Add σReparam to DeepNorm | 100L-100L Transformer · WMT'17 English-German newstest2017 | Test BLEU: 29.2$_{0.0}$ -> 28.0$_{0.1}$ | Hurt |
| Lower the global attention temperature to below 0.15 during warmup (ablation) | ViT-B/16 · ImageNet1k | Training stability | Hurt |
| Increase global attention temperature to 100 (ablation) | ViT-B/16 · ImageNet1k | Performance: 100 | Hurt |
| Use σReparam with trunc_normal(.01) initialization instead of trunc_normal(.02) (ablation) | SimCLR ViT-B/16 · ImageNet1k linear probe | Training stability: 0.01 -> .02 | Hurt |
| Replace the SimCLR baseline with σReparam | ViT-B/16 · ImageNet1k linear probe, 300 epochs | Top-1 accuracy: 72.4 -> 73.7 | Helped |
Closest papers by meaning
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape (ICML 2024 · similarity 0.88)
- From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics (NeurIPS 2025 · similarity 0.87)
- Non-asymptotic Convergence of Training Transformers for Next-token Prediction (NeurIPS 2024 · similarity 0.87)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.87)
- Small-scale proxies for large-scale Transformer training instabilities (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.