Research map / Architectures and efficiency / Attention and transformers
Scaling Vision Transformers to 22 Billion Parameters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add query/key LayerNorm (QK normalization) (ablation) | 8B parameter ViT · training stability | training divergence | Helped |
| Keep learning rate at 1e-3 with QK normalization instead of reducing it with scale (ablation) | ViT models up to ViT-L · ImageNet/scale study | stability and quality with scale | Helped |
| Omit biases on QKV projections and LayerNorms | ViT-22B · training efficiency | accelerator utilization: +3% utilization | Helped |
| Parallel attention and MLP layers | ViT-22B · training efficiency | training speed: 15% speedup cited from PaLM | Helped |
| Scale ViT from ViT-e to ViT-22B with linear probing on ImageNet | frozen ViT features · ImageNet-1k linear probe | top-1 accuracy: 89.26 -> 89.51 | Helped |
| Scale to 22B parameters for ObjectNet zero-shot via LiT | LiT text tower with frozen ViT · ObjectNet zero-shot | top-1 accuracy: 84.9 -> 87.6 (LiT-e to LiT-22B) | Helped |
| Scale ViT to 22B for few-shot semantic segmentation | ADE20k with 1/16 training data · ADE20k 1/16 data | mIoU: +8.6 over DeiT-III Large, +2.3 over ViT-G | Helped |
| Scale ViT to 22B for full ADE20k training data | ADE20k full training set · ADE20k 1/1 data | mIoU: 54.9 vs ViT-G 55.6 | Hurt |
| Scale to 22B with frozen features for monocular depth | Waymo depth estimation · Waymo Open depth, DPT and linear decoder | MSE: DPT 0.024 -> 0.021; linear 0.053 -> 0.039 (vs ViT-e) | Helped |
| Scale to 22B for frozen-backbone video classification | Kinetics 400 and Moments in Time · video classification frozen backbone | top-1 accuracy: +1.5 Kinetics, +1.3 Moments in Time over ViT-e | Helped |
| Scale to 22B and use ECE calibration with fine-tuning on ImageNet | ViT fine-tuned on ImageNet 384px · ImageNet-C Plex benchmark | ECE: 0.05 (ViT-L/32) -> 0.01 (ViT-22B) | Helped |
| Distill from ViT-22B teacher into ViT-B/16 student | ViT-B/16 · ImageNet 384px | top-1 accuracy: 86.7 -> 88.6 | Helped |
Closest papers by meaning
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.88)
- Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness (AAAI 2025 · similarity 0.88)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.87)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.87)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.87)
- Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.