Research map / Architectures and efficiency / CNN design
Early Convolutions Help Transformers See Better
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace patchify stem with convolutional stem | ViT-1GF · ImageNet-1k | top-1 error: 27.7 -> 24.7 | Helped |
| replace patchify stem with convolutional stem | ViT-18GF · ImageNet-1k | top-1 error: 17.9 -> 17.0 | Helped |
| replace patchify stem with convolutional stem | ViT-36GF · ImageNet-21k pretrain | top-1 error: 15.1 -> 14.2 | Helped |
| replace patchify stem with convolutional stem (ablation) | ViT-4GF · ImageNet-1k 50ep | AdamW-SGD gap: 5.3 -> 0.7 | Helped |
| replace patchify stem with convolutional stem | ViT-1GF · ImageNet-1k 50ep | gap to 400-epoch error: 10% -> about 6% | Helped |
| increase patchify layer size in stem from 2 to 16 (ablation) | ViT_C-4GF · ImageNet-1k 50ep | top-1 error: 24.3 -> 26.2 | Hurt |
| add BN and ReLU after patchify stem (ablation) | ViT_P-4GF · ImageNet-1k | top-1 error | No clear effect |
| replace BN with LN in conv stem (ablation) | ViT_C-4GF · ImageNet-1k | top-1 error | No clear effect |
Closest papers by meaning
- Scaled ReLU Matters for Training Vision Transformers (AAAI 2022 · similarity 0.92)
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.89)
- Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words (ICLR 2024 · similarity 0.89)
- Your ViT is Secretly an Image Segmentation Model (CVPR 2025 · similarity 0.89)
- Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization (NeurIPS 2024 · similarity 0.88)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.