Research map / Architectures and efficiency / Attention and transformers
Scalable Vision Transformers With Hierarchical Pooling
Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, Jianfei Cai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hierarchical pooling (1 stage) to DeiT-Ti | DeiT-Ti · ImageNet | Top-1 accuracy: 72.20 -> 69.64 | Hurt |
| add hierarchical pooling (1 stage) to DeiT-S | DeiT-S · ImageNet | Top-1 accuracy: 79.80 -> 78.00 | Hurt |
| scale up embedding dim and heads with saved FLOPs, 4 pooling stages | HVT-Ti · ImageNet | Top-1 accuracy: 72.20 -> 75.23 | Improved |
| remove class token and predict from average-pooled tokens (ablation) | DeiT-Ti · CIFAR-100 | Top-1 accuracy: 64.49 -> 65.93 | Improved |
| remove class token and predict from average-pooled tokens (ablation) | DeiT-Ti · ImageNet | Top-1 accuracy: 72.20 -> 72.42 | Improved |
| use max pooling instead of average pooling (ablation) | HVT-S-4 · CIFAR-100 | Top-1 accuracy: 70.38 -> 75.43 | Improved |
| use max pooling instead of convolution downsampling (ablation) | HVT-S-4 · CIFAR-100 | Top-1 accuracy: 69.75 -> 75.43 | Improved |
| increase pooling stages from 0 to 4 (ablation) | HVT-S · CIFAR-100 | Top-1 accuracy: 71.99 -> 75.43 | Improved |
| increase pooling stages from 0 to 4 (ablation) | HVT-S · ImageNet | Top-1 accuracy: 80.39 -> 75.23 | Hurt |
| increase Transformer blocks from 12 to 24 (ablation) | HVT-S-4 · CIFAR-100 | Top-1 accuracy: 75.43 -> 75.04 | No clear effect |
| increase attention heads from 3 to 16 (ablation) | HVT-Ti-4 · CIFAR-100 | Top-1 accuracy: 69.51 -> 76.30 | Improved |
| increase image resolution from 160 to 384 (ablation) | HVT-S-4 · CIFAR-100 | Top-1 accuracy: 73.84 -> 76.31 | Improved |
Closest papers by meaning
- Convolutional Embedding Makes Hierarchical Vision Transformer Stronger (ECCV 2022 · similarity 0.89)
- Scaling Vision Transformers (CVPR 2022 · similarity 0.89)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.88)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.88)
- Data-independent Module-aware Pruning for Hierarchical Vision Transformers (ICLR 2024 · similarity 0.87)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
