Research map / Architectures and efficiency / Attention and transformers
Spectral Conditioning of Attention Improves Transformer Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes spectral conditioned attention, which adds fixed identity-scaled correction terms (λI, λ=10) to the query, key and value projection weights to reduce attention Jacobian condition number, and reports small accuracy gains across ViT/DeiT/Swin/XCiT/DaViT, COCO detection, LRA and crammed BERT.
Problem
Transformer attention Jacobians may be ill-conditioned, which can hinder gradient-based optimization; the conditioning of attention layers has been largely unexamined.
Approach
Derives an upper bound on the attention Jacobian condition number in terms of the condition numbers of W_Q, W_K, W_V, and shows that adding a fixed diagonal correction λI_k (λ≥2) reduces those condition numbers without SVD, yielding drop-in "spectral conditioned attention".
Key results
- On ImageNet-1k, spectral conditioning improved ViT-B top-1 from 80.7 to 81.7 and DaViT-B from 84.3 to 84.9, with gains in all five vision models.
- On COCO with XCiT-S Mask R-CNN, AP^b rose from 44.9 to 45.6 and AP^m from 40.1 to 40.5.
- On crammed BERT, GLUE average rose from 78.6 to 79.4 with the spectrally conditioned variant winning on every task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spectral conditioning to attention | ViT-B, DeiT-B, Swin-B, XCiT-M, and DaViT-B · ImageNet-1k | Top-1% accuracy: 80.7 ($\pm$0.41) -> 81.7 ($\pm$0.38); 81.6 ($\pm$0.30) -> 82.6 ($\pm$0.32); 83.4 ($\pm$0.28) -> 84.1 ($\pm$0.25); 82.6 ( | Helped |
| Add spectral conditioning to attention | XCiT-S · COCO 2017 | AP^b: 44.9 ($\pm$0.33) -> 45.6 ($\pm$0.31) | Helped |
| Add spectral conditioning to attention | XCiT-S · COCO 2017 | AP^b_50: 66.1 ($\pm$0.31) -> 66.7 ($\pm$0.36) | Helped |
| Add spectral conditioning to attention | XCiT-S · COCO 2017 | AP^b_75: 48.9 ($\pm$0.35) -> 49.6 ($\pm$0.32) | Helped |
| Add spectral conditioning to attention | XCiT-S · COCO 2017 | AP^m: 40.1 ($\pm$0.29) -> 40.5 ($\pm$0.33) | Helped |
| Add spectral conditioning to attention | XCiT-S · COCO 2017 | AP^m_50: 63.1 ($\pm$0.31) -> 63.4 ($\pm$0.28) | Helped |
| Add spectral conditioning to attention | XCiT-S · COCO 2017 | AP^m_75: 42.8 ($\pm$0.34) -> 43.3 ($\pm$0.32) | Helped |
| Add spectral conditioning to attention | Nyströmformer · LRA | accuracy: ListOps: 37.1 ($\pm$0.21) -> 37.8 ($\pm$0.23); Text: 63.8 ($\pm$0.24) -> 64.8 ($\pm$0.20); Retrieval: 79.8 ($\pm$0.26) - | Helped |
| Add spectral conditioning to attention | Crammed BERT · GLUE | Avg.: 78.6 -> 79.4 | Helped |
| Set the correction scale to λ = 10 rather than λ = 2, 4, 6, 8, 12, 14, or 16 (ablation) | ViT-B · ImageNet-1k | accuracy: 80.8; 81.1; 81.4; 81.6; 81.7; 81.7; 81.6; 81.5 | Helped |
| Remove layer normalization while retaining spectral conditioning (λ = 10) (ablation) | ViT-B, DeiT-B, Swin-B, XCiT-M, and DaViT-B · ImageNet-1k | Top-1% accuracy: 81.7 ($\pm$0.38) -> 79.8 ($\pm$0.31); 82.6 ($\pm$0.32) -> 80.4 ($\pm$0.33); 84.1 ($\pm$0.25) -> 80.5 ($\pm$0.23); 83.5 ( | Hurt |
| Use the SVD-derived correction (Thm. 3.5) instead of the implementation-friendly correction (Thm. 3.8) (ablation) | Vision transformers · ImageNet-1k | training time: ViT-B: 29:33 -> 41:38; DeiT-B: 26:24 -> 37:54; Swin-B: 53:26 -> 68:32; XCiT-M: 91:18 -> 109:12; DaViT-M: 75:28 -> 93:12 | Hurt |
Surprising findings
- Removing layer normalization while keeping spectral conditioning degraded accuracy well below the original baseline (e.g., ViT-B 79.8 vs 80.7), which the authors attribute to large weight magnitudes.
Limitations
- The method optimizes an upper bound on the Jacobian condition number, not the Jacobian condition number itself.
- Experiments are limited to models up to about 100M parameters, so scaling behavior is unverified.
- The theory is derived for self-attention, with extension to other attention variants argued but not proven.
Experimental setup
Datasets: ImageNet-1k COCO 2017 Long-Range Arena The Pile GLUE
Benchmarks: ImageNet-1k COCO 2017 Long-Range Arena GLUE
Metrics: top-1 accuracy AP^b AP^m accuracy average GLUE score condition number minimum singular value
Baselines: ViT-B DeiT-B Swin-B XCiT-M DaViT-B Nystromformer Crammed BERT
Builds on: Swin Transformer XCiT Nystromformer Mask R-CNN
Closest papers by meaning
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.88)
- Consensus Is All You Get: The Role of Attention in Transformers (ICML 2025 · similarity 0.87)
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.87)
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.