Research map / Architectures and efficiency / Attention and transformers
CoAtNet: Marrying Convolution and Attention for All Data Sizes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add relative attention to standard attention (ablation) | CoAtNet-2 · ImageNet-1K, 224² | accuracy: 83.8 -> 84.1 | Helped |
| Add relative attention to standard attention (ablation) | CoAtNet-2 · ImageNet-1K, 384² | accuracy: 85.3 -> 85.7 | Helped |
| Add relative attention to standard attention (ablation) | CoAtNet-3 · ImageNet-21K → ImageNet-1K, 384² | finetune accuracy: 87.4 -> 87.9 | Helped |
| Shift blocks from S2 MBConv to S3 Transformer (ablation) | CoAtNet-2 · ImageNet-1K | top-1 accuracy: 84.1 -> 83.4 | Hurt |
| Shift blocks from S3 Transformer to S2 MBConv (ablation) | CoAtNet-2 · ImageNet-1K | top-1 accuracy: 84.1 -> 84.0 | Hurt |
| Increase attention head size from 32 to 64 (ablation) | CoAtNet-2 · ImageNet-1K, 224² | top-1 accuracy: 84.1 -> 83.9 | Hurt |
| Replace BatchNorm with LayerNorm in MBConv (ablation) | CoAtNet-2 · ImageNet-1K, 224² | top-1 accuracy: 84.1 -> 84.1 | No clear effect |
| Replace BatchNorm with LayerNorm in MBConv (ablation) | CoAtNet-3 · ImageNet-21K → ImageNet-1K, 384² | top-1 accuracy: 87.9 -> 87.8 | Hurt |
| Perform down-sampling in the depthwise convolution instead of the first convolution (ablation) | CoAtNet-0 · ImageNet-1K, 224² | top-1 accuracy: 81.6 -> 82.0 | Helped |
| Perform down-sampling in the depthwise convolution instead of the first convolution (ablation) | CoAtNet-1 · ImageNet-1K, 224² | top-1 accuracy: 83.3 -> 83.5 | Helped |
| Perform down-sampling in the depthwise convolution instead of the first convolution (ablation) | CoAtNet-2 · ImageNet-1K, 224² | top-1 accuracy: 84.1 -> 84.1 | No clear effect |
| Apply RandAugment during ImageNet-21K pre-training (ablation) | CoAtNet-4 · ImageNet-21K → ImageNet-1K, 384² | top-1 accuracy: 87.9 -> 88.3 | Helped |
Closest papers by meaning
- AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation (NeurIPS 2024 · similarity 0.86)
- LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate (NeurIPS 2024 · similarity 0.86)
- SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (ICML 2024 · similarity 0.86)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.86)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.86)
- LSNet: See Large, Focus Small (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.