Research map / Architectures and efficiency / Attention and transformers
Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training
Seyed Morteza Emadi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace delayed scaling with geometry-aware scaling computed from weights | FP8 attention scaling · first forward pass after loading pretrained weights | overflowing layers: Llama-2-70B 80/80 -> 0/80 | Improved |
| replace delayed scaling with geometry-aware scaling | FP8 attention scaling · checkpoint resumption (first 10 steps) | overflow steps: delayed: 4 steps on GPT-2 XL -> 0 | Improved |
| replace delayed scaling with geometry-aware scaling | FP8 attention scaling · 100x learning rate spike | overflow steps: delayed: 3 to 5 overflow steps -> 0 | Improved |
| use conservative spectral alpha (0.03) instead of delayed scaling (ablation) | Llama-2-13B MMLU STEM fine-tuning · MMLU STEM test | accuracy (%): 32.6 -> 28.7 | Hurt |
| add auto-alpha calibration from burn-in slack ratios (ablation) | geometry-aware scaling on Llama-2-13B · MMLU STEM test | accuracy (%): 28.7 -> 32.7 | Improved |
| add auto-alpha calibration (ablation) | geometry-aware scaling on Llama-2-13B · MMLU STEM fine-tuning | FP8 utilization (%): 0.5% -> 34.9% | Improved |
| auto-alpha vs delayed scaling downstream accuracy | Llama-2-13B · MMLU STEM test | accuracy (%): 32.6 -> 32.7 | No clear effect |
| use implicit GQA power iteration instead of key expansion (ablation) | geometry-aware scaling on Mistral-7B · forward pass timing | overhead (%): -5.3% vs delayed scaling | Improved |
| use interaction matrix norm instead of product of separate norms (ablation) | logit bound · theory | bound tightness | Improved |
Closest papers by meaning
- Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.90)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.87)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.87)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.86)
- Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs (NeurIPS 2025 · similarity 0.86)
- Scaling Limits of Long-Context Transformers (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
