AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Visformer: The Vision-Friendly Transformer

Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, Qi Tian

ICCV 2021 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 296 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add global average pooling instead of classification token (ablation)DeiT-S · ImageNet (base setting)top-1 accuracy: 64.17 -> 69.44Improved
add step-wise patch embeddings instead of large patch embedding (ablation)Net1 · ImageNet (base setting)top-1 accuracy: 69.81 -> 73.01Improved
add step-wise patch embeddings instead of large patch embedding (ablation)Net1 · ImageNet (elite setting)top-1 accuracy: 80.16 -> 81.35Improved
add stage-wise design (ablation)Net2 · ImageNet (base vs elite setting)top-1 accuracy: base 73.01 -> 75.76; elite 81.35 -> 80.19Mixed
add batch norm instead of layer norm (ablation)Net3 · ImageNet (elite setting)top-1 accuracy: 80.19 -> 80.97Improved
add 3x3 convolutions in feed-forward blocks (ablation)Net4 · ImageNet (base vs elite setting)top-1 accuracy: base 76.49 -> 77.37; elite 80.97 -> 80.15Mixed
add position embedding (ablation)DeiT-S · ImageNet (elite setting)top-1 accuracy: elite performance decreases significantly by 3.95% when removedImproved
add position embedding (ablation)Net5 · ImageNet (elite setting)top-1 accuracy: 79.86 -> 80.15Improved
add self-attention instead of convolution (ablation)Net6 · ImageNet (elite setting)top-1 accuracy: 79.01 -> 79.86Improved
add self-attention in stage 1 instead of bottleneck blocks (ablation)Net5 · ImageNet (elite setting)top-1 accuracy: 80.15 -> 80.13No clear effect
add self-attention in stage 3 instead of bottleneck blocks (ablation)Net5 · ImageNet (elite setting)top-1 accuracy: 79.59 -> 80.15Improved
add bottleneck blocks with 3x3 convolutions in stage 1 only (ablation)Net4 · ImageNet (elite setting)top-1 accuracy: 80.97 -> 81.10Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy