Research map / Architectures and efficiency / CNN design
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train ViT-H/14 on JFT-300M rather than use the strongest reported CNN baseline | ImageNet · ImageNet | accuracy: 87.54 -> 88.55 | Helped |
| Pre-train ViT-H/14 on JFT-300M rather than use BiT-L | ImageNet ReaL · ImageNet ReaL | accuracy: 90.54 -> 90.72 | Helped |
| Scale ViT pre-training from ImageNet to ImageNet-21k to JFT-300M | ViT transfer · ImageNet | accuracy: 77.91 -> 83.97 -> 84.15 | Helped |
| Remove positional embeddings (ablation) | ViT-B/16 · ImageNet 5-shot linear | accuracy: 0.64206 -> 0.61382 | Hurt |
| Replace 1-D positional embeddings with 2-D positional embeddings (ablation) | ViT-B/16 · ImageNet 5-shot linear | accuracy: 0.64206 -> 0.64001 | No clear effect |
| Replace 1-D positional embeddings with relative positional embeddings (ablation) | ViT-B/16 · ImageNet 5-shot linear | accuracy: 0.64206 -> 0.64032 | No clear effect |
| Add positional embeddings at every layer rather than only after the stem (ablation) | ViT-B/16 · ImageNet 5-shot linear | accuracy: 0.64206 -> 0.63964 | No clear effect |
| Add shared positional embeddings at every layer rather than only after the stem (ablation) | ViT-B/16 · ImageNet 5-shot linear | accuracy: 0.64206 -> 0.64292 | No clear effect |
| Use masked patch prediction for self-supervised pre-training rather than training from scratch | ViT-B/16 · ImageNet | accuracy: 79.9%; a significant improvement of 2% | Helped |
| Use masked patch prediction instead of supervised pre-training | ViT-B/16 · ImageNet | accuracy: still 4% behind supervised pre-training | Hurt |
| Use 15% rather than 50% patch corruption (ablation) | Masked patch prediction · ImageNet few-shot metrics | accuracy | Hurt |
| Use L2 full-patch regression rather than mean-color prediction (ablation) | Masked patch prediction · ImageNet few-shot performance | accuracy | Hurt |
Closest papers by meaning
- Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (NeurIPS 2021 · similarity 0.91)
- CMT: Convolutional Neural Networks Meet Vision Transformers (CVPR 2022 · similarity 0.91)
- Intriguing Properties of Vision Transformers (NeurIPS 2021 · similarity 0.91)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.89)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.89)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.