AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, Jianfeng Gao

ICCV 2021 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs · 382 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use relative positional bias instead of absolute 2-D positional embedding (ablation)ViL-Small · ImageNet-1K valtop-1 accuracy (%): 82.0 -> 82.4Improved
replace full attention with Vision Longformer attention (ablation)multi-scale ViT Small-4stage 1-2-8-1 · ImageNet-1K valtop-1 accuracy (%): 81.91 -> 81.99No clear effect
add LayerNorm after the patch embedding (ablation)DeiT-Tiny (CLS feature) · ImageNet-1K valtop-1 accuracy (%): 72.2 -> 72.91Improved
add LayerNorm after the patch embedding (ablation)DeiT-Small (CLS feature) · ImageNet-1K valtop-1 accuracy (%): 79.8 -> 80.33Improved
use average-pooled feature instead of CLS token feature (ablation)ViL-1,2,8,1-APE Small · ImageNet-1K valtop-1 accuracy (%): 81.65 -> 81.99Improved
use multi-scale stacked structure instead of flat ViT (ablation)DeiT-Small / full attention · ImageNet-1K valtop-1 accuracy (%): 79.9 -> 81.7Improved
use Vision Longformer attention instead of SRA/64 in all stages (ablation)multi-scale ViT Small · COCO Mask R-CNN 1xbox AP: 36.4 -> 42.9Improved
use Vision Longformer attention instead of pure global attention in all stages (ablation)multi-scale ViT Small · COCO Mask R-CNN 1xbox AP: 34.8 -> 42.9Improved
use Vision Longformer attention instead of Performer in all stages (ablation)multi-scale ViT Small · COCO Mask R-CNN 1xbox AP: 36.1 -> 42.9Improved
use ViL in the first two stages and full attention in the last two (Partial ViL) (ablation)multi-scale ViT Small · COCO Mask R-CNN 1xbox AP: 42.9 -> 43.3No clear effect
keep full attention in last two stages (Partial Linformer) (ablation)Linformer Tiny-4stage 1-1-9-1 · ImageNet-1K valtop-1 accuracy (%): 74.71 -> 75.56Improved
keep full attention in last two stages (Partial Performer) (ablation)Performer Tiny-4stage 1-1-9-1 · ImageNet-1K valtop-1 accuracy (%): 71.12 -> 75.34Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy