AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

What Makes for Hierarchical Vision Transformer?

Y.K. Fang, Xinggang Wang, Rui Wu, Wenyu Liu

TPAMI 2023 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 21 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace window MHSA with linear mapping (ablation)Swin (shift) · ImageNet-1k (200 epochs)top-1 accuracy (%): 80.5 -> 79.7Hurt
replace window MHSA with linear mapping (ablation)Shuffle Transformer · ImageNet-1k (200 epochs)top-1 accuracy (%): 80.6 -> 79.6Hurt
replace window MHSA with linear mapping (ablation)MSG Transformer · ImageNet-1k (200 epochs)top-1 accuracy (%): 80.4 -> 79.5Hurt
change cross-window communication (shift, shuffle, MSG) (ablation)Linear-Tiny · ImageNet-1k (300 epochs)top-1 accuracy (%): 80.5 / 80.6 / 80.4No clear effect
use depth-wise linear (separate weights per group) instead of shared linear (ablation)Linear-Tiny · ImageNet-1k (200 epochs)top-1 accuracy (%): 78.8 -> 78.9No clear effect
use MLP with GELU instead of linear mapping (ablation)Linear-Tiny · ImageNet-1k (200 epochs)top-1 accuracy (%): 78.8 -> 79.0No clear effect
make model wider (width 112) (ablation)Linear-Tiny · ImageNet-1k (200 epochs)top-1 accuracy (%): 78.8 -> 79.8Improved
make model deeper (width 64, depth 2,4,22,4) (ablation)Linear-Tiny · ImageNet-1k (200 epochs)top-1 accuracy (%): 78.8 -> 79.7Improved
use 96 groups instead of 32 in linear layers (ablation)Linear-Tiny · ImageNet-1k (200 epochs)top-1 accuracy (%): 78.8 -> 78.4Hurt
use 8 groups instead of 32 in linear layers (ablation)Linear-Tiny · ImageNet-1k (200 epochs)top-1 accuracy (%): 78.8 -> 78.6Hurt
scale width from Tiny to Small to BaseSwin Linear · ImageNet-1k (300 epochs)top-1 accuracy (%): 80.5 -> 82.0 -> 82.5Improved
use Linear-Tiny instead of Swin-Mixer-T/D6Swin Linear-Tiny · ImageNet-1ktop-1 accuracy (%): 79.7 -> 80.5Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy