Research map / Architectures and efficiency / Attention and transformers
What Makes for Hierarchical Vision Transformer?
Y.K. Fang, Xinggang Wang, Rui Wu, Wenyu Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace window MHSA with linear mapping (ablation) | Swin (shift) · ImageNet-1k (200 epochs) | top-1 accuracy (%): 80.5 -> 79.7 | Hurt |
| replace window MHSA with linear mapping (ablation) | Shuffle Transformer · ImageNet-1k (200 epochs) | top-1 accuracy (%): 80.6 -> 79.6 | Hurt |
| replace window MHSA with linear mapping (ablation) | MSG Transformer · ImageNet-1k (200 epochs) | top-1 accuracy (%): 80.4 -> 79.5 | Hurt |
| change cross-window communication (shift, shuffle, MSG) (ablation) | Linear-Tiny · ImageNet-1k (300 epochs) | top-1 accuracy (%): 80.5 / 80.6 / 80.4 | No clear effect |
| use depth-wise linear (separate weights per group) instead of shared linear (ablation) | Linear-Tiny · ImageNet-1k (200 epochs) | top-1 accuracy (%): 78.8 -> 78.9 | No clear effect |
| use MLP with GELU instead of linear mapping (ablation) | Linear-Tiny · ImageNet-1k (200 epochs) | top-1 accuracy (%): 78.8 -> 79.0 | No clear effect |
| make model wider (width 112) (ablation) | Linear-Tiny · ImageNet-1k (200 epochs) | top-1 accuracy (%): 78.8 -> 79.8 | Improved |
| make model deeper (width 64, depth 2,4,22,4) (ablation) | Linear-Tiny · ImageNet-1k (200 epochs) | top-1 accuracy (%): 78.8 -> 79.7 | Improved |
| use 96 groups instead of 32 in linear layers (ablation) | Linear-Tiny · ImageNet-1k (200 epochs) | top-1 accuracy (%): 78.8 -> 78.4 | Hurt |
| use 8 groups instead of 32 in linear layers (ablation) | Linear-Tiny · ImageNet-1k (200 epochs) | top-1 accuracy (%): 78.8 -> 78.6 | Hurt |
| scale width from Tiny to Small to Base | Swin Linear · ImageNet-1k (300 epochs) | top-1 accuracy (%): 80.5 -> 82.0 -> 82.5 | Improved |
| use Linear-Tiny instead of Swin-Mixer-T/D6 | Swin Linear-Tiny · ImageNet-1k | top-1 accuracy (%): 79.7 -> 80.5 | Improved |
Closest papers by meaning
- Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows (ICCV 2021 · similarity 0.90)
- Learned Queries for Efficient Local Attention (CVPR 2022 · similarity 0.88)
- Neighborhood Attention Transformer (CVPR 2023 · similarity 0.88)
- Cross Aggregation Transformer for Image Restoration (NeurIPS 2022 · similarity 0.88)
- Less Is More: Pay Less Attention in Vision Transformers (AAAI 2022 · similarity 0.88)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
