Research map / Architectures and efficiency / CNN design
Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SOPE (sequential overlapping patch embedding) (ablation) | DeiT-T · CIFAR-100 | accuracy (%): 67.59 -> 73.68 | Helped |
| add DAFF (dynamic aggregation feed forward) (ablation) | DeiT-T · CIFAR-100 | accuracy (%): 67.59 -> 79.47 | Helped |
| add head tokens (ablation) | DeiT-T · CIFAR-100 | accuracy (%): 67.59 -> 69.10 | Helped |
| add head tokens on top of SOPE+DAFF (no abs PE) (ablation) | DHVT-T · CIFAR-100 | accuracy (%): 80.35 -> 80.85 | Helped |
| remove absolute positional embedding (ablation) | DeiT-T · CIFAR-100 | accuracy (%): 67.59 -> 58.72 | Hurt |
| remove absolute positional embedding (ablation) | DeiT-T with DAFF · CIFAR-100 | accuracy (%): 79.47 -> 79.75 | Helped |
| aggregate and re-calibrate all tokens instead of only class token in DAFF (ablation) | DHVT-T · CIFAR-100 | accuracy (%): 80.85 -> 78.34 | Hurt |
| add shortcut alongside DWCONV in DAFF (ablation) | DHVT-T · CIFAR-100 | accuracy (%): 80.14 -> 80.85 | Helped |
| add affine transforms in SOPE (ablation) | DHVT-T · CIFAR-100 | accuracy (%): 80.72 -> 80.90 | Helped |
| use BatchNorm instead of LayerNorm in SOPE and DAFF (ablation) | DHVT-T · CIFAR-100 | accuracy (%): 78.90 -> 80.98 | Helped |
| split class token from FFN processing (ablation) | ViT · CIFAR-100 | accuracy (%): 67.59 -> 68.76 | Helped |
| number of heads 4 vs 3 (ablation) | DHVT-T · CIFAR-100 | accuracy (%): 80.92 -> 80.98 | No clear effect |
Closest papers by meaning
- Do Vision Transformers See Like Convolutional Neural Networks? (NeurIPS 2021 · similarity 0.91)
- ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases (ICML 2021 · similarity 0.90)
- The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy (CVPR 2022 · similarity 0.90)
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.88)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.88)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.