AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention

Haiquan Qiu, Quanming Yao

ICLR 2026 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 11 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
disable tiling by setting block size equal to sequence length (ablation)flash attention 2 · GPT-2S BF16 OpenWebTexttraining stabilityNo clear effect
replace flash attention with standard attention in layer 2 (ablation)GPT-2 flash attention BF16 · GPT-2S BF16 OpenWebTexttraining stabilityImproved
use flash attention only in layer 2 (ablation)GPT-2 flash attention BF16 · GPT-2S BF16 OpenWebTexttraining stabilityHurt
compute delta as rowsum(dP*P) instead of rowsum(dO*O) (ablation)flash attention backward · GPT-2S BF16 OpenWebTexttraining stabilityImproved
recompute O as PV in FP32 for delta in backward (ablation)flash attention backward · GPT-2S BF16 OpenWebTexttraining stabilityImproved
compute O in FP32 in forward pass (ablation)flash attention forward · GPT-2S BF16 OpenWebTexttraining stabilityImproved
compute O in high precision only for outlier heads 1,7,8,9,11,12 (ablation)flash attention · GPT-2S BF16 OpenWebTexttraining stabilityImproved
modify softmax normalization to avoid P entries exactly equal to 1 (beta=2)flash attention · GPT-2S with AdamW and Muon, GPT-2M AdamWloss explosionImproved
use standard BF16 flash attentionGPT-2 · GPT-2S BF16 OpenWebTextvalidation lossHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy