Research map / Architectures and efficiency / Attention and transformers
Green Hierarchical Vision Transformer for Masked Image Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visible-patch Group Window Attention, DP-based grouping, and sparse convolution | Swin-B hierarchical MIM · ImageNet-1K val | top-1 accuracy (%): 84.0 -> 83.8 | No clear effect |
| Use visible-patch Group Window Attention, DP-based grouping, and sparse convolution | Swin-B hierarchical MIM · MS-COCO val | box AP: 50.4 -> 50.0 | No clear effect |
| Use visible-patch Group Window Attention, DP-based grouping, and sparse convolution | Swin-B hierarchical MIM · ImageNet-1K pre-training | total pre-training hours: 1609 -> 887 | Helped |
| Increase pre-training window size from 7×7 to 14×14 (ablation) | Ours · ImageNet-1K | top-1 accuracy (%): 83.2% -> 83.4% | Helped |
| Increase pre-training window size from 7×7 to 14×14 (ablation) | Ours · larger-window ablation | GPU memory: 121.6G -> 148.9G | Hurt |
| Increase pre-training window size from 7×7 to 14×14 (ablation) | Ours · larger-window ablation | training time: 1.1h -> 1.2h | Hurt |
| Use a single group sized to the sum of visible tokens rather than DP-optimized grouping (ablation) | Group Window Attention · group attention FLOPs | FLOPs: 52.3M -> 52.7M; 52.7M -> 201.3M | Hurt |
| Remove the DP solver while using group size max_i w_i (ablation) | Group Window Attention · group attention FLOPs | FLOPs: 64.5M -> 137.5M; 59.7M -> 113.3M; 62.5M -> 75.7M | Hurt |
| Vary mask ratio from 0.5 to 0.85 (ablation) | Ours · ImageNet-1K fine-tuning | accuracy | No clear effect |
| Use fewer decoder blocks (ablation) | Ours · ImageNet-1K fine-tuning | accuracy | Helped |
Closest papers by meaning
- HiViT: A Simpler and More Efficient Design of Hierarchical Vision Transformer (ICLR 2023 · similarity 0.90)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.87)
- Data-independent Module-aware Pruning for Hierarchical Vision Transformers (ICLR 2024 · similarity 0.87)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.86)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.86)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.