Research map / Architectures and efficiency / RNNs and normalization
Transformers learn through gradual rank increase
Enric Boix-Adserà, Etai Littwin, Emmanuel Abbe, Samy Bengio, Joshua M. Susskind
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a practical ViT with Adam and track attention-weight perturbations | ViT · CIFAR-10 | stable rank of attention-weight perturbations | Improved |
| Train a practical ViT with Adam and track attention-weight perturbations | ViT · CIFAR-100 | stable rank of attention-weight perturbations | Improved |
| Train a practical ViT with Adam and track attention-weight perturbations | ViT-Base/16 · ImageNet | stable rank of attention-weight perturbations | Improved |
| Train a language transformer and track attention-weight perturbations | GPT-2 · Wikitext-103 | stable rank of attention-weight perturbations | Improved |
| Train with SGD rather than Adam | ViT · CIFAR-10 and CIFAR-100 | stable rank of attention-weight perturbations | Improved |
| Use small initialization and diagonal attention weights, training only attention-head weights | Diagonal attention-head transformer · Diagonal attention-head toy task | rank increase per training stage | Improved |
| Reduce initialization scale (ablation) | ViT · CIFAR-10 | stable rank after training | Improved |
| Use default rather than smaller initialization scale (ablation) | ViT · CIFAR-10 | visibility of incremental rank increase | Hurt |
| Apply the practical-model analysis without diagonal-weight or small-initialization assumptions | Vision and language transformers · CIFAR-10, CIFAR-100, ImageNet, and Wikitext-103 | gradual increase in stable rank | Improved |
| Compare the learned perturbation spectrum with the initialized weight spectrum | ViT · CIFAR-10 | rank of learned attention-weight perturbation | Improved |
Closest papers by meaning
- From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics (NeurIPS 2025 · similarity 0.87)
- Abrupt Learning in Transformers: A Case Study on Matrix Completion (NeurIPS 2024 · similarity 0.85)
- Linear attention is (maybe) all you need (to understand Transformer optimization) (ICLR 2024 · similarity 0.85)
- Small-scale proxies for large-scale Transformer training instabilities (ICLR 2024 · similarity 0.85)
- Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models (ICLR 2025 · similarity 0.84)
- Infinite Limits of Multi-head Transformer Dynamics (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
