AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Incorporating Convolution Designs Into Visual Transformers

Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, Wei Wu

ICCV 2021 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 656 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add I2T stem (k7s2 conv, k3s2 maxpool, BN) instead of patch embedding (ablation)DeiT-T · ImageNet val (DeiT-T ablation)Top-1 accuracy: 72.2 -> 73.4Improved
add maxpool after the I2T convolution (ablation)I2T · ImageNet val (DeiT-T ablation)Top-1 accuracy: 72.9 -> 73.4 (with BN); k7s4 conv alone 71.4Improved
add BatchNorm to I2T (ablation)I2T · ImageNet val (DeiT-T ablation)Top-1 accuracy: 72.9 -> 73.4Improved
use a single k7s4 conv without maxpool as I2T (ablation)DeiT-T · ImageNet val (DeiT-T ablation)Top-1 accuracy: 72.2 -> 71.4Hurt
use two k3s2 convs as I2T (ablation)DeiT-T · ImageNet val (DeiT-T ablation)Top-1 accuracy: 72.2 -> 70.4Hurt
add LeFF with 1x1 kernel (ablation)DeiT-T · ImageNet val (DeiT-T ablation)Top-1 accuracy: 72.2 -> 70.3Hurt
add LeFF with 3x3 depth-wise conv (no BN) (ablation)DeiT-T · ImageNet val (DeiT-T ablation)Top-1 accuracy: 72.2 -> 72.7Improved
add BatchNorm to LeFF 3x3 (ablation)LeFF · ImageNet val (DeiT-T ablation)Top-1 accuracy: 72.7 -> 74.3Improved
enlarge LeFF kernel from 3x3 to 5x5 (with BN) (ablation)LeFF · ImageNet val (DeiT-T ablation)Top-1 accuracy: 74.3 -> 74.4No clear effect
add LCA module (ablation)DeiT-T · ImageNet val (DeiT-T ablation)Top-1 accuracy: 72.2 -> 72.8Improved
replace DeiT-T with CeiT-T at 100 epochsDeiT-T · ImageNet val (100 epochs)Top-1 accuracy: 65.3 -> 72.2Improved
replace DeiT-S with CeiT-S at 100 epochsDeiT-S · ImageNet val (100 epochs)Top-1 accuracy: 74.5 -> 78.9Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy