AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Three Things Everyone Should Know about Vision Transformers

Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, Hervé Jégou

ECCV 2022 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 151 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add LayerScale and train 400 epochs instead of 300ViT-L/16 · ImageNet-1k valtop-1 accuracy: 83.0 -> 84.0Improved
fine-tune only MHSA weights instead of all weights at 384 resolutionViT-B · ImageNet-1k valtop-1 accuracy: 84.3 -> 84.3No clear effect
fine-tune only FFN instead of MHSA weights at 384 resolution (ablation)ViT-S · ImageNet-1k valtop-1 accuracy: 82.5 -> 82.2Hurt
fine-tune only attention weights on small datasetsViT-S · CARStop-1 accuracy: 89.7 -> 89.8Improved
fine-tune only attention weights on large datasetViT-S · INAT-18top-1 accuracy: 68.0 -> 60.6Hurt
fine-tune only attention weights with larger modelViT-L · INAT-18top-1 accuracy: 75.9 -> 75.3Mixed
fine-tune only FFN instead of attention on small dataset (ablation)ViT-B · CARStop-1 accuracy: 92.8 -> 88.9Hurt
replace linear stem with hMLP stem (BN) in BeiT pre-training (ablation)ViT-B12 · ImageNet-1k val (BeiT+FT)top-1 accuracy: 83.05 -> 83.43Improved
replace linear stem with hMLP stem (BN) in supervised training (ablation)ViT-B12 · ImageNet-1k valtop-1 accuracy: 82.20 -> 82.54Improved
replace linear stem with conv stem in BeiT pre-training (ablation)ViT-B12 · ImageNet-1k val (BeiT+FT)top-1 accuracy: 83.05 -> 83.04No clear effect
use local transformer stem in BeiT pre-training (ablation)ViT-B12 · ImageNet-1k val (BeiT+FT)top-1 accuracy: 83.05 -> 82.38Hurt
add GELU without normalization to linear stem (ablation)ViT-B12 · ImageNet-1k valtop-1 accuracy: 82.20 -> 81.55Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy