Research map / Architectures and efficiency / Attention and transformers
Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Pale-T with pale-shaped attention instead of CSWin-T | ImageNet-1K classification · ImageNet-1K validation | Top-1 accuracy (%): 82.7 -> 83.4 | Helped |
| Use Pale-S instead of CSWin-S | ImageNet-1K classification · ImageNet-1K validation | Top-1 accuracy (%): +0.7% | Helped |
| Use pale-shaped attention backbone Pale-T instead of CSWin-T | UperNet · ADE20K validation, single-scale | mIoU: +1.1% | Helped |
| Use Pale-T instead of CSWin-T | Mask R-CNN 1x · COCO val2017 | box AP: +0.7 | Helped |
| Increase pale size from 1 to 7 across stages (ablation) | Pale-T · ImageNet-1K | Top-1 accuracy (%): 82.4 -> 83.4 | Helped |
| Increase pale size from 1 to 7 across stages (ablation) | Pale-T · ADE20K | SS mIoU (%): 47.9 -> 50.4 | Helped |
| Increase pale size from 7 to 9 across stages (ablation) | Pale-T · ImageNet-1K | Top-1 accuracy (%): 83.4 -> 83.3 | Hurt |
| Increase pale size from 7 to 9 across stages (ablation) | Pale-T · ADE20K | SS mIoU (%): 50.4 -> 50.6 | Helped |
| Replace PS-Attention with axial attention (ablation) | Pale-T · ImageNet-1K | Top-1 accuracy (%): 82.4 -> 83.4 | Hurt |
| Replace PS-Attention with cross-shaped attention (ablation) | Pale-T · ADE20K | SS mIoU (%): 49.0 -> 50.4 | Hurt |
| Use parallel rather than sequential PS-Attention (ablation) | Pale-T · COCO val2017 | box AP: 46.9 -> 47.4 | Helped |
| Replace parallel PS-Attention with vanilla PS-Attention (ablation) | Pale-T · COCO val2017 | mask AP: 42.7 -> 42.3 | Hurt |
Closest papers by meaning
- Focal Attention for Long-Range Interactions in Vision Transformers (NeurIPS 2021 · similarity 0.89)
- Slide-Transformer: Hierarchical Vision Transformer With Local Self-Attention (CVPR 2023 · similarity 0.89)
- CSWin Transformer: A General Vision Transformer Backbone With Cross-Shaped Windows (CVPR 2022 · similarity 0.89)
- Training Object Detectors From Scratch: An Empirical Study in the Era of Vision Transformer (CVPR 2022 · similarity 0.88)
- PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer (ICLR 2025 · similarity 0.87)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.