Research map / Architectures and efficiency / Attention and transformers
Slide-Transformer: Hierarchical Vision Transformer With Local Self-Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace early-stage PVT attention with Slide Attention | PVT-T · ImageNet-1K val | Top-1 accuracy: 75.1 -> 78.0 | Helped |
| Replace early-stage Swin attention with Slide Attention | Swin-T · ImageNet-1K val | Top-1 accuracy: 81.3 -> 82.3 | Helped |
| Replace early-stage PVT attention with Slide Attention | PVT-T + SemanticFPN · ADE20K val | mIoU: 36.57 -> 38.43 | Helped |
| Replace early-stage Swin attention with Slide Attention | Swin-T + UperNet · ADE20K val | mIoU: 44.51 -> 45.67 | Helped |
| Replace early-stage PVT attention with Slide Attention | PVT-T + RetinaNet · COCO val, 1x | AP: 36.7 -> 40.1 | Helped |
| Replace early-stage Swin attention with Slide Attention | Swin-T + Cascade Mask R-CNN · COCO val, 3x | APᵇ: 50.4 -> 51.1 | Helped |
| Use Slide Attention instead of NAT local attention | Swin-T setting · ImageNet-1K val | accuracy: 81.8 -> 82.3 | Helped |
| Use Slide Attention in stages 1 and 2 only rather than stage 1 only (ablation) | Swin-T · ImageNet-1K val | accuracy: 81.8 -> 82.3 | Helped |
| Add Slide Attention to stage 3 (ablation) | Swin-T with Slide Attention in stages 1–2 · ImageNet-1K val | accuracy: 82.3 -> 82.2 | Hurt |
| Add Slide Attention to stage 4 (ablation) | Swin-T with Slide Attention in stages 1–3 · ImageNet-1K val | accuracy: 82.2 -> 81.3 | Hurt |
| Expand Slide Attention from the first two stages to all four stages (ablation) | Swin-T · ImageNet-1K val | accuracy: 82.3 -> 81.3 | Hurt |
| Increase the local-attention window size beyond k=3 (ablation) | Slide Attention · Swin-T and Swin-S | accuracy | No clear effect |
Closest papers by meaning
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.89)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.88)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.88)
- LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition (AAAI 2024 · similarity 0.88)
- Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.