Research map / Detection and segmentation / CNN features
Focal Attention for Long-Range Interactions in Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Swin window attention with focal attention | Swin-Tiny · ImageNet-1K val | top-1 accuracy (%): 81.2 -> 82.2 | Helped |
| replace Swin window attention with focal attention | Swin-Small/Base · COCO Mask R-CNN 1x | box AP: 46.5 -> 47.4 | Helped |
| add fine-grained local surrounding tokens to window attention (ablation) | Focal-Tiny-Window · COCO Mask R-CNN 1x | box AP: 38.8 -> 43.9 | Helped |
| add coarse-grained global tokens to window attention (ablation) | Focal-Tiny-Window · ImageNet-1K val | top-1 accuracy (%): 80.1 -> 81.6 | Helped |
| combine local and global interactions (ablation) | Focal-Tiny · ImageNet-1K val | top-1 accuracy (%): 81.6 -> 82.2 | Helped |
| add window shift (ablation) | Focal-Tiny · ImageNet-1K val | top-1 accuracy (%): 82.2 -> 81.9 | Hurt |
| add window shift (ablation) | Focal-Tiny · COCO Mask R-CNN 1x | box AP: 44.8 -> 44.9 | No clear effect |
| add window shift (ablation) | Swin-Tiny · ImageNet-1K val | top-1 accuracy (%): 80.2 -> 81.2 | Helped |
| increase window size from 7 to 14 (ablation) | Focal-Tiny · ImageNet-1K val | top-1 accuracy (%): 82.2 -> 82.3 | Helped |
| increase window size from 7 to 14 (ablation) | Focal-Tiny · COCO Mask R-CNN 1x | box AP: 44.9 -> 45.5 | Helped |
| increase stage-3 depth from 2 to 6 layers (ablation) | Focal-Tiny · ImageNet-1K val | top-1 accuracy (%): 79.9 -> 82.2 | Helped |
| focal attention gives comparable accuracy with fewer layers (ablation) | Focal-Tiny 2-2-4-2 vs Swin-Tiny 2-2-6-2 · ImageNet-1K val | top-1 accuracy (%): 81.2 -> 81.4 | Helped |
Closest papers by meaning
- Slide-Transformer: Hierarchical Vision Transformer With Local Self-Attention (CVPR 2023 · similarity 0.90)
- RegionViT: Regional-to-Local Attention for Vision Transformers (ICLR 2022 · similarity 0.90)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.88)
- PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer (ICLR 2025 · similarity 0.88)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.