Research map / Architectures and efficiency / Attention and transformers
DaViT: Dual Attention Vision Transformers
Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, Lu Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dual attention block in stage 1 (ablation) | window-attention transformer · ImageNet-1K (28.3M model) | Top-1: 81.1 -> 81.7 | Improved |
| add dual attention block in stage 2 (ablation) | window-attention transformer · ImageNet-1K (28.3M model) | Top-1: 81.1 -> 82.2 | Improved |
| add dual attention block in stage 3 (ablation) | window-attention transformer · ImageNet-1K (28.3M model) | Top-1: 81.1 -> 82.1 | Improved |
| add dual attention block in stage 4 (ablation) | window-attention transformer · ImageNet-1K (28.3M model) | Top-1: 81.1 -> 81.9 | Improved |
| add dual attention in all four stages (ablation) | window-attention transformer · ImageNet-1K (28.3M model) | Top-1: 81.1 -> 82.8 | Improved |
| put channel attention before window attention instead of after (ablation) | DaViT-Tiny · ImageNet-1K (28.3M model) | Top-1: 82.8 -> 82.6 | Hurt |
| arrange window and channel attention in parallel instead of sequentially (ablation) | DaViT-Tiny · ImageNet-1K (28.3M model) | Top-1: 82.8 -> 82.6 | Hurt |
| replace half of DeiT attention blocks with channel attention (ablation) | DeiT-Tiny · ImageNet-1K | Top-1: 72.2 -> 74.9 | Improved |
| replace half of DeiT attention blocks with channel attention (ablation) | DeiT-Base · ImageNet-1K | Top-1: 81.8 -> 82.3 | Improved |
| use dual attention instead of pure window attention without FFN (ablation) | Tiny w/o FFN · ImageNet-1K | Top-1: 79.1 -> 80.8 | Improved |
| use dual attention instead of pure channel attention without FFN (ablation) | Tiny w/o FFN · ImageNet-1K | Top-1: 79.3 -> 80.8 | Improved |
| use DaViT instead of Swin at 2x fewer third-stage layers (ablation) | Tiny-scale, depths · ImageNet-1K | Top-1: 81.2 -> 82.8 | Improved |
Closest papers by meaning
- MaxViT: Multi-axis Vision Transformer (ECCV 2022 · similarity 0.91)
- ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases (ICML 2021 · similarity 0.90)
- RegionViT: Regional-to-Local Attention for Vision Transformers (ICLR 2022 · similarity 0.90)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.89)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.87)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
