Research map / Architectures and efficiency / Attention and transformers
Multi-Head Normalization for Wide Vision Transformers
Guoyizhe Wei, Feng Wang, Alan Yuille, Rama Chellappa
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace global RMSNorm with MH-RMSNorm on ViT-H | ViT-H · ImageNet-1K | top-1 accuracy: 85.2 -> 85.5 | Improved |
| replace global RMSNorm with MH-RMSNorm on ViT-B | ViT-B · ImageNet-1K | top-1 accuracy: 83.8 -> 83.9 | No clear effect |
| replace global RMSNorm with MH-RMSNorm on ViT-H | ViT-H · ADE20K | mIoU: 50.6 -> 51.0 | Improved |
| replace RMSNorm with MH-RMSNorm in DiT-XL/2 (7M steps) | DiT-XL/2 · ImageNet-256 | FID: 2.20 -> 1.95 | Improved |
| replace RMSNorm with MH-RMSNorm in DiT-XL/2 (800k steps) | DiT-XL/2 · ImageNet-256 | FID: 13.8 -> 13.0 | Improved |
| partition width-4096 features into 8 to 32 chunks instead of global RMSNorm (ablation) | ViT width 4096 · ImageNet-1K | top-1 accuracy: 61.1 -> 84.4 | Improved |
| use too-small chunks (d=64 instead of 256) (ablation) | MH-RMSNorm width 4096 · ImageNet-1K | top-1 accuracy: 84.4 -> 83.5 | Hurt |
| use MH-LayerNorm instead of MH-RMSNorm (ablation) | MH-Norm width 4096 · ImageNet-1K | top-1 accuracy: 84.4 -> 84.2 | Hurt |
| use head-wise instead of shared affine parameters in MH-RMSNorm (ablation) | MH-RMSNorm width 4096 · ImageNet-1K | top-1 accuracy: 84.0 -> 84.4 | Improved |
| apply MH-RMSNorm to both attention and MLP branches instead of attention only (ablation) | ViT width 2048 · ImageNet-1K | top-1 accuracy: 84.0 -> 84.2 | Improved |
| apply MH-RMSNorm to all layers instead of only deep layers 9-12 (ablation) | ViT width 2048 · ImageNet-1K | top-1 accuracy: 83.9 -> 84.2 | Improved |
| apply MH-RMSNorm to only shallow layers 1-4 (ablation) | ViT width 2048 · ImageNet-1K | top-1 accuracy: 82.2 -> 83.1 | Mixed |
Closest papers by meaning
- Scaling Vision Transformers to 22 Billion Parameters (ICML 2023 · similarity 0.87)
- The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy (CVPR 2022 · similarity 0.87)
- Data-independent Module-aware Pruning for Hierarchical Vision Transformers (ICLR 2024 · similarity 0.86)
- Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (NeurIPS 2024 · similarity 0.86)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.86)
- Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
