Research map / Architectures and efficiency / Attention and transformers
CvT: Introducing Convolutions to Vision Transformers
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, Lei Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add position embedding (ablation) | DeiT-S · ImageNet-1k val | top-1 accuracy (%): 78.0 -> 79.8 | Improved |
| add position embedding (every stage) (ablation) | CvT-13 · ImageNet-1k val | top-1 accuracy (%): 81.6 -> 81.5 | No clear effect |
| add convolutional token embedding (no position embedding) (ablation) | CvT-13 · ImageNet-1k val | top-1 accuracy (%): 80.7 -> 81.6 | Improved |
| add convolutional token embedding (with position embedding) (ablation) | CvT-13 · ImageNet-1k val | top-1 accuracy (%): 81.1 -> 81.4 | Improved |
| add convolutional projection in all three stages (ablation) | CvT-13 · ImageNet-1k val | top-1 accuracy (%): 80.6 -> 81.6 | Improved |
| add convolutional projection in stage 1 only (ablation) | CvT-13 · ImageNet-1k val | top-1 accuracy (%): 80.6 -> 80.8 | Improved |
| add convolutional projection in stages 1 and 2 (ablation) | CvT-13 · ImageNet-1k val | top-1 accuracy (%): 80.6 -> 81.0 | Improved |
| use stride 2 instead of 1 for key/value projection (ablation) | CvT-13 · ImageNet-1k val | top-1 accuracy (%): 82.3 -> 81.6 | Mixed |
| neural architecture search over stride and MLP ratio | CvT-13 · ImageNet-1k val | top-1 accuracy (%): 81.6 -> 82.2 | Improved |
| scale from CvT-21 to wide CvT-W24 with ImageNet-22k pretraining | CvT · ImageNet-1k val | top-1 accuracy (%): 84.9 -> 87.7 | Improved |
| raise input resolution from 224 to 384 | CvT-21 · ImageNet-1k val | top-1 accuracy (%): 82.5 -> 83.3 | Improved |
| ImageNet-22k pretraining | CvT-13 at 384 · ImageNet-1k val | top-1 accuracy (%): 83.0 -> 83.3 | No clear effect |
Closest papers by meaning
- CMT: Convolutional Neural Networks Meet Vision Transformers (CVPR 2022 · similarity 0.93)
- A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking (TPAMI 2024 · similarity 0.93)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ICLR 2021 · similarity 0.92)
- Intriguing Properties of Vision Transformers (NeurIPS 2021 · similarity 0.92)
- ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases (ICML 2021 · similarity 0.92)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
