Research map / Architectures and efficiency / Attention and transformers
Three Things Everyone Should Know about Vision Transformers
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, Hervé Jégou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LayerScale and train 400 epochs instead of 300 | ViT-L/16 · ImageNet-1k val | top-1 accuracy: 83.0 -> 84.0 | Improved |
| fine-tune only MHSA weights instead of all weights at 384 resolution | ViT-B · ImageNet-1k val | top-1 accuracy: 84.3 -> 84.3 | No clear effect |
| fine-tune only FFN instead of MHSA weights at 384 resolution (ablation) | ViT-S · ImageNet-1k val | top-1 accuracy: 82.5 -> 82.2 | Hurt |
| fine-tune only attention weights on small datasets | ViT-S · CARS | top-1 accuracy: 89.7 -> 89.8 | Improved |
| fine-tune only attention weights on large dataset | ViT-S · INAT-18 | top-1 accuracy: 68.0 -> 60.6 | Hurt |
| fine-tune only attention weights with larger model | ViT-L · INAT-18 | top-1 accuracy: 75.9 -> 75.3 | Mixed |
| fine-tune only FFN instead of attention on small dataset (ablation) | ViT-B · CARS | top-1 accuracy: 92.8 -> 88.9 | Hurt |
| replace linear stem with hMLP stem (BN) in BeiT pre-training (ablation) | ViT-B12 · ImageNet-1k val (BeiT+FT) | top-1 accuracy: 83.05 -> 83.43 | Improved |
| replace linear stem with hMLP stem (BN) in supervised training (ablation) | ViT-B12 · ImageNet-1k val | top-1 accuracy: 82.20 -> 82.54 | Improved |
| replace linear stem with conv stem in BeiT pre-training (ablation) | ViT-B12 · ImageNet-1k val (BeiT+FT) | top-1 accuracy: 83.05 -> 83.04 | No clear effect |
| use local transformer stem in BeiT pre-training (ablation) | ViT-B12 · ImageNet-1k val (BeiT+FT) | top-1 accuracy: 83.05 -> 82.38 | Hurt |
| add GELU without normalization to linear stem (ablation) | ViT-B12 · ImageNet-1k val | top-1 accuracy: 82.20 -> 81.55 | Hurt |
Closest papers by meaning
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ICLR 2021 · similarity 0.91)
- Transformer Fusion with Optimal Transport (ICLR 2024 · similarity 0.88)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.88)
- SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (ICML 2024 · similarity 0.88)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.88)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
