Research map / Architectures and efficiency / Attention and transformers
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding
Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, Jianfeng Gao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use relative positional bias instead of absolute 2-D positional embedding (ablation) | ViL-Small · ImageNet-1K val | top-1 accuracy (%): 82.0 -> 82.4 | Improved |
| replace full attention with Vision Longformer attention (ablation) | multi-scale ViT Small-4stage 1-2-8-1 · ImageNet-1K val | top-1 accuracy (%): 81.91 -> 81.99 | No clear effect |
| add LayerNorm after the patch embedding (ablation) | DeiT-Tiny (CLS feature) · ImageNet-1K val | top-1 accuracy (%): 72.2 -> 72.91 | Improved |
| add LayerNorm after the patch embedding (ablation) | DeiT-Small (CLS feature) · ImageNet-1K val | top-1 accuracy (%): 79.8 -> 80.33 | Improved |
| use average-pooled feature instead of CLS token feature (ablation) | ViL-1,2,8,1-APE Small · ImageNet-1K val | top-1 accuracy (%): 81.65 -> 81.99 | Improved |
| use multi-scale stacked structure instead of flat ViT (ablation) | DeiT-Small / full attention · ImageNet-1K val | top-1 accuracy (%): 79.9 -> 81.7 | Improved |
| use Vision Longformer attention instead of SRA/64 in all stages (ablation) | multi-scale ViT Small · COCO Mask R-CNN 1x | box AP: 36.4 -> 42.9 | Improved |
| use Vision Longformer attention instead of pure global attention in all stages (ablation) | multi-scale ViT Small · COCO Mask R-CNN 1x | box AP: 34.8 -> 42.9 | Improved |
| use Vision Longformer attention instead of Performer in all stages (ablation) | multi-scale ViT Small · COCO Mask R-CNN 1x | box AP: 36.1 -> 42.9 | Improved |
| use ViL in the first two stages and full attention in the last two (Partial ViL) (ablation) | multi-scale ViT Small · COCO Mask R-CNN 1x | box AP: 42.9 -> 43.3 | No clear effect |
| keep full attention in last two stages (Partial Linformer) (ablation) | Linformer Tiny-4stage 1-1-9-1 · ImageNet-1K val | top-1 accuracy (%): 74.71 -> 75.56 | Improved |
| keep full attention in last two stages (Partial Performer) (ablation) | Performer Tiny-4stage 1-1-9-1 · ImageNet-1K val | top-1 accuracy (%): 71.12 -> 75.34 | Improved |
Closest papers by meaning
- ResFormer: Scaling ViTs With Multi-Resolution Training (CVPR 2023 · similarity 0.91)
- Scaling Vision Transformers (CVPR 2022 · similarity 0.90)
- A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking (TPAMI 2024 · similarity 0.89)
- LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition (AAAI 2024 · similarity 0.88)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
- ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
