Research map / Architectures and efficiency / Attention and transformers
Incorporating Convolution Designs Into Visual Transformers
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, Wei Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add I2T stem (k7s2 conv, k3s2 maxpool, BN) instead of patch embedding (ablation) | DeiT-T · ImageNet val (DeiT-T ablation) | Top-1 accuracy: 72.2 -> 73.4 | Improved |
| add maxpool after the I2T convolution (ablation) | I2T · ImageNet val (DeiT-T ablation) | Top-1 accuracy: 72.9 -> 73.4 (with BN); k7s4 conv alone 71.4 | Improved |
| add BatchNorm to I2T (ablation) | I2T · ImageNet val (DeiT-T ablation) | Top-1 accuracy: 72.9 -> 73.4 | Improved |
| use a single k7s4 conv without maxpool as I2T (ablation) | DeiT-T · ImageNet val (DeiT-T ablation) | Top-1 accuracy: 72.2 -> 71.4 | Hurt |
| use two k3s2 convs as I2T (ablation) | DeiT-T · ImageNet val (DeiT-T ablation) | Top-1 accuracy: 72.2 -> 70.4 | Hurt |
| add LeFF with 1x1 kernel (ablation) | DeiT-T · ImageNet val (DeiT-T ablation) | Top-1 accuracy: 72.2 -> 70.3 | Hurt |
| add LeFF with 3x3 depth-wise conv (no BN) (ablation) | DeiT-T · ImageNet val (DeiT-T ablation) | Top-1 accuracy: 72.2 -> 72.7 | Improved |
| add BatchNorm to LeFF 3x3 (ablation) | LeFF · ImageNet val (DeiT-T ablation) | Top-1 accuracy: 72.7 -> 74.3 | Improved |
| enlarge LeFF kernel from 3x3 to 5x5 (with BN) (ablation) | LeFF · ImageNet val (DeiT-T ablation) | Top-1 accuracy: 74.3 -> 74.4 | No clear effect |
| add LCA module (ablation) | DeiT-T · ImageNet val (DeiT-T ablation) | Top-1 accuracy: 72.2 -> 72.8 | Improved |
| replace DeiT-T with CeiT-T at 100 epochs | DeiT-T · ImageNet val (100 epochs) | Top-1 accuracy: 65.3 -> 72.2 | Improved |
| replace DeiT-S with CeiT-S at 100 epochs | DeiT-S · ImageNet val (100 epochs) | Top-1 accuracy: 74.5 -> 78.9 | Improved |
Closest papers by meaning
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ICLR 2021 · similarity 0.92)
- Convolutional Embedding Makes Hierarchical Vision Transformer Stronger (ECCV 2022 · similarity 0.91)
- CMT: Convolutional Neural Networks Meet Vision Transformers (CVPR 2022 · similarity 0.90)
- CvT: Introducing Convolutions to Vision Transformers (ICCV 2021 · similarity 0.90)
- Do Vision Transformers See Like Convolutional Neural Networks? (NeurIPS 2021 · similarity 0.90)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
