Research map / Architectures and efficiency / Attention and transformers
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove relative position embeddings, use absolute (ablation) | MViTv2-L trained with MAE · ImageNet-1K | top-1 accuracy: 85.6 vs 85.6 baseline | No clear effect |
| replace convs with maxpools (ablation) | Hiera-L MAE · ImageNet-1K | top-1 accuracy: drops accuracy by over 1% | Hurt |
| delete stride=1 maxpools (ablation) | Hiera-L MAE · ImageNet-1K | top-1 accuracy: 84.4 -> 85.4, speeds up 22% | Helped |
| set maxpool kernel size equal to stride (no overlap) (ablation) | Hiera-L MAE · ImageNet-1K | top-1 accuracy: 85.4 -> 85.7, ~20% faster on images | Helped |
| remove attention residual for Q pooling (ablation) | Hiera-L MAE · ImageNet-1K | top-1 accuracy: 85.7 -> 85.6 | No clear effect |
| replace KV pooling with mask unit attention (ablation) | Hiera-L MAE · Kinetics-400 | top-1 accuracy: 85.5 -> 85.5, throughput up to 32% on video | No clear effect |
| replace KV pooling with mask unit attention (ablation) | Hiera-L MAE · ImageNet-1K | top-1 accuracy: 85.6 -> 85.6, throughput 374 -> 531 im/s | No clear effect |
| multi-scale decoder (fuse all stages) (ablation) | Hiera-L MAE pretraining decoder · ImageNet-1K | top-1 accuracy: 85.0 -> 85.6 | Helped |
| multi-scale decoder (fuse all stages) (ablation) | Hiera-L MAE pretraining decoder · Kinetics-400 | top-1 accuracy: 83.8 -> 85.5 | Helped |
| drop path during MAE pretraining (0.2 vs 0.0) (ablation) | Hiera-L MAE pretraining · ImageNet-1K | top-1 accuracy: 85.2 -> 85.6 | Helped |
| drop path during MAE pretraining (0.2 vs 0.0) (ablation) | Hiera-L MAE pretraining · Kinetics-400 | top-1 accuracy: 84.5 -> 85.5 | Helped |
| longer MAE pretraining (400 to 1600 epochs) (ablation) | Hiera-L MAE · ImageNet-1K | top-1 accuracy: 85.6 -> 86.1 | Helped |
Closest papers by meaning
- Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding (AAAI 2022 · similarity 0.88)
- Data-independent Module-aware Pruning for Hierarchical Vision Transformers (ICLR 2024 · similarity 0.87)
- Learning Hierarchical Image Segmentation For Recognition and By Recognition (ICLR 2024 · similarity 0.87)
- Win-Win: Training High-Resolution Vision Transformers from Two Windows (ICLR 2024 · similarity 0.87)
- Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers (CVPR 2024 · similarity 0.86)
- Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learner (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.