AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Rethinking Spatial Dimensions of Vision Transformers

Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, Seong Joon Oh

ICCV 2021 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs · 750 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add ResNet-style spatial reduction to ResNet50 (ablation)ResNet50 · ImageNet-1ktraining loss / validation accuracyImproved
add pooling layers (PiT) to ViT, vanilla trainingViT-Ti · ImageNet-1k valtop-1 accuracy: 68.7% -> 71.3%Improved
add pooling layers (PiT) to ViT, vanilla trainingViT-S · ImageNet-1k valtop-1 accuracy: 68.7% -> 73.3%Improved
add pooling layers (PiT) to ViT, vanilla trainingViT-B · ImageNet-1k valtop-1 accuracy: 69.3% -> 76.1%Improved
add pooling layers (PiT) to ViT, CutMix trainingViT-B · ImageNet-1k valtop-1 accuracy: 75.3% -> 79.9%Improved
add pooling layers (PiT) to ViT, DeiT trainingViT-Ti · ImageNet-1k valtop-1 accuracy: 72.2% -> 73.0%Improved
add pooling layers (PiT) to ViT, DeiT trainingViT-B · ImageNet-1k valtop-1 accuracy: 81.8% -> 82.0%Improved
add pooling layers (PiT) to ViT with distillationViT-Ti · ImageNet-1k valtop-1 accuracy: 74.5% -> 74.6%No clear effect
add pooling layers (PiT) to ViT, 1000 epoch trainingViT-Ti · ImageNet-1k valtop-1 accuracy: 76.6% -> 76.4%Hurt
add pooling layers (PiT) to ViT, 384x384 fine-tuneViT-B · ImageNet-1k valtop-1 accuracy: 83.1% -> 83.0%No clear effect
use PiT-S backbone instead of ViT-SDeformable DETR · COCO val2017AP: 36.9 -> 39.4Improved
use PiT-S backbone instead of ViT-SImageNet-A · ImageNet-Aaccuracy: 19.1 -> 21.7Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy