Research map / Architectures and efficiency / Attention and transformers
Taming Curvature: Architecture Warm-up for Stable Transformer Training
Sameera Ramasinghe, Thalaiyasingam Ajanthan, Hadi Mohaghegh Dolatabadi, Chamin P Hewa Koneputugodage, Gil Avraham, Violetta Shevchenko, Yan Zuo, Karol Pajak, Alexander Long
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add architecture warm-up (start at half depth, unlock layers gradually) | Llama-style 1B Transformer · FineWeb perplexity (peak LR 8e-3) | validation perplexity: QK-Norm 49.88 -> 25.02; baseline diverged | Improved |
| add architecture warm-up | Llama-style 1B Transformer · OLMo-Mix perplexity | validation perplexity: all baselines diverged -> 18.54 | Improved |
| add architecture warm-up versus Softcap | Llama-style 1B Transformer · DCLM perplexity | validation perplexity: 43.38 -> 22.64 | Improved |
| use QK-Clip stabilization | Llama-style 1B Transformer · FineWeb/DCLM/OLMo-Mix | divergence | Hurt |
| architecture warm-up vs QK-Norm at Chinchilla-optimal compute | 1B, 25B tokens · FineWeb | validation perplexity: 20.28 -> 18.35 | Improved |
| increase depth from 8 to 16/32 layers at LR 8e-3 without warm-up (ablation) | Transformer · FineWeb curvature and stability | lambda_max and loss spikes | Hurt |
| remove LR warmup and use architecture warm-up only (ablation) | 1B Transformer · FineWeb | convergence and stability | No clear effect |
| change unlock spacing of layers to zero (ablation) | Architecture warm-up · FineWeb | convergence | No clear effect |
| warm-start the power iteration with previous top eigenvector (ablation) | online Hessian eigenvalue estimation · 4-layer Transformer | relative error of curvature | Improved |
| sweep peak LR 3e-3 to 8e-3 with architecture warm-up (ablation) | Transformer · FineWeb | stability | Improved |
Closest papers by meaning
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.88)
- Taming Transformer Without Using Learning Rate Warmup (ICLR 2025 · similarity 0.88)
- Global Convergence in Training Large-Scale Transformers (NeurIPS 2024 · similarity 0.87)
- Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity (NeurIPS 2026 · similarity 0.85)
- Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency (NeurIPS 2025 · similarity 0.85)
- Linear attention is (maybe) all you need (to understand Transformer optimization) (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
