Research map / Architectures and efficiency / CNN design
Can CNNs Be More Robust Than Transformers?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ViT-style 16x16 patchify stem (ablation) | ResNet variants (DW, Inverted-DW, Up-, Down-Inverted-DW) · ImageNet-C, Stylized-ImageNet, ImageNet-R, ImageNet-Sketch | robustness accuracy/error: at least 1.2% robustness boost; up to 6.6% on Stylized-ImageNet | Helped |
| Increase patchify stem patch size from 8 to 16 (ablation) | ResNet variants with ViT-style patchify stem · ImageNet clean accuracy | clean accuracy (IN): clean accuracy drops e.g. 79.9 to 78.5 for ResNet-DW | Hurt |
| Use a convolutional stem instead of ViT-style patchify stem (ablation) | ResNet variants · ImageNet-R / Stylized-ImageNet | robustness accuracy | Hurt |
| Enlarge depth-wise kernel size (3 to 11 or 13) (ablation) | ResNet depth-wise variants · Stylized-ImageNet, ImageNet-C, ImageNet-R, ImageNet-Sketch | robustness accuracy | Helped |
| Remove activation and normalization layers, keeping one norm and one activation (Norm1Act2) (ablation) | ResNet-Up-Inverted-DW · ImageNet-R | accuracy (IN-R): 42.9 to 44.2 (+1.3 with single activation/norm choice); up to 47.2 with norm1 act1 variant | Helped |
| Remove two of three norm and activation layers per block (keep one of each) (ablation) | ResNet variants · ImageNet-1k clean accuracy | accuracy (IN): e.g. ResNet-DW 79.7 to 79.4 and ResNet-Inverted-DW 80.0 to 80.7 in the best-placement rows | Mixed |
| Combine patchify stem, large kernel, and reduced norm/activation layers (ablation) | ResNet variants (Robust-ResNet) · Stylized-ImageNet, ImageNet-C, ImageNet-R, ImageNet-Sketch | robustness accuracy: Robust-ResNet beats DeiT-S on all four OOD benchmarks (e.g., S-IN 20.2 vs 16.2; IN-C 40.9 vs 42.8 for Up-Inverted-DW) | Helped |
| Add structural re-parameterization to Robust-ResNet (ablation) | Robust-ResNet-DW and Down-Inverted-DW (DeiT-S scale) · ImageNet-R / Stylized-ImageNet | robustness accuracy: Down-Inverted-DW IN-R 46.0 to 49.0; Up-Inverted-DW S-IN 20.2 to 18.5 | Mixed |
| Knowledge distillation from DeiT-S to Robust-ResNet students (ablation) | Robust-ResNet students · ImageNet-R / Stylized-ImageNet | robustness accuracy: Robust-ResNet-DW S-IN 10.2 (ResNet-DW) to 20.1 | Helped |
Closest papers by meaning
- Are Transformers more robust than CNNs? (NeurIPS 2021 · similarity 0.95)
- Revealing the Dark Secrets of Extremely Large Kernel ConvNets on Robustness (ICML 2024 · similarity 0.88)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.88)
- Towards Understanding and Improving Adversarial Robustness of Vision Transformers (CVPR 2024 · similarity 0.87)
- Vision Transformers Beat WideResNets on Small Scale Datasets Adversarial Robustness (AAAI 2025 · similarity 0.87)
- Curved Representation Space of Vision Transformers (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.