Research map / Architectures and efficiency / CNN design
Scaled ReLU Matters for Training Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace patchify-stem with a conv-stem containing BN+ReLU | DeiT-Small · ImageNet-1k val | Top-1 accuracy: 79.8 -> 81.9 | Helped |
| Add scaled ReLU (BN+ReLU) throughout the conv-stem (ablation) | DeiT-Small · ImageNet-1k val | Top-1 accuracy: 79.8 -> 81.9 | Helped |
| Remove ReLU from the conv-stem (ablation) | DeiT-Small · ImageNet-1k val | training stability | Hurt |
| Remove BN while retaining convolution and ReLU (ablation) | DeiT-Small · ImageNet-1k val | Top-1 accuracy: 81.9 -> 81.5 | Hurt |
| Remove BN and ReLU from the conv-stem (ablation) | DeiT-Small · ImageNet-1k val | training stability | Hurt |
| Increase warmup to 20 epochs after removing BN and ReLU (ablation) | DeiT-Small conv-stem · ImageNet-1k val | Top-1 accuracy: crash -> 80.0 | Helped |
| Replace per-block ReLU with one ReLU after projection (ablation) | DeiT-Small conv-stem · ImageNet-1k val | Top-1 accuracy: 81.9 -> 79.9 | Hurt |
| Change conv-stem stride from (2,1,1,8) to (2,2,2,2) (ablation) | DeiT-Small · ImageNet-1k val | Top-1 accuracy: 81.9 -> 81.0 | Hurt |
| Replace ReLU with GELU in the conv-stem (ablation) | DeiT-Small · ImageNet-1k val | Top-1 accuracy: 81.9 -> 81.7 | Hurt |
| Replace GELU with ReLU in the feed-forward network (ablation) | DeiT-Small · ImageNet-1k val | Top-1 accuracy: 81.9 -> 80.3 | Hurt |
| Remove LayerNorm from the feed-forward network (ablation) | DeiT-Small and VOLO-d1 · ImageNet-1k val | training stability | Hurt |
| Replace ffn MLP with Conv1D+BN+GELU (ablation) | DeiT-Small · ImageNet-1k val | Top-1 accuracy: 81.9 -> 81.7 | Hurt |
Closest papers by meaning
- Early Convolutions Help Transformers See Better (NeurIPS 2021 · similarity 0.92)
- Scaling Vision Transformers (CVPR 2022 · similarity 0.89)
- Auto-scaling Vision Transformers without Training (ICLR 2022 · similarity 0.89)
- A General and Efficient Training for Transformer via Token Expansion (CVPR 2024 · similarity 0.87)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.87)
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.