AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training

Seyed Morteza Emadi

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace delayed scaling with geometry-aware scaling computed from weightsFP8 attention scaling · first forward pass after loading pretrained weightsoverflowing layers: Llama-2-70B 80/80 -> 0/80Improved
replace delayed scaling with geometry-aware scalingFP8 attention scaling · checkpoint resumption (first 10 steps)overflow steps: delayed: 4 steps on GPT-2 XL -> 0Improved
replace delayed scaling with geometry-aware scalingFP8 attention scaling · 100x learning rate spikeoverflow steps: delayed: 3 to 5 overflow steps -> 0Improved
use conservative spectral alpha (0.03) instead of delayed scaling (ablation)Llama-2-13B MMLU STEM fine-tuning · MMLU STEM testaccuracy (%): 32.6 -> 28.7Hurt
add auto-alpha calibration from burn-in slack ratios (ablation)geometry-aware scaling on Llama-2-13B · MMLU STEM testaccuracy (%): 28.7 -> 32.7Improved
add auto-alpha calibration (ablation)geometry-aware scaling on Llama-2-13B · MMLU STEM fine-tuningFP8 utilization (%): 0.5% -> 34.9%Improved
auto-alpha vs delayed scaling downstream accuracyLlama-2-13B · MMLU STEM testaccuracy (%): 32.6 -> 32.7No clear effect
use implicit GQA power iteration instead of key expansion (ablation)geometry-aware scaling on Mistral-7B · forward pass timingoverhead (%): -5.3% vs delayed scalingImproved
use interaction matrix norm instead of product of separate norms (ablation)logit bound · theorybound tightnessImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy