Research map / Architectures and efficiency / Attention and transformers
MOAT: Alternating Mobile Convolution and Attention Brings Strong Vision Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Transformer MLP with MBConv, including added normalization and activation (ablation) | MOAT block · ImageNet-1K, MOAT-0, input 224² | top-1 accuracy: 82.6 -> 82.9 | Helped |
| Insert depthwise convolution into the MLP without adding the extra normalization and activation (ablation) | Transformer block · ImageNet-1K, MOAT-0, input 224² | top-1 accuracy: 82.6 -> 81.7 | Hurt |
| Reorder MBConv before attention (ablation) | MOAT block · ImageNet-1K, MOAT-0, input 224² | top-1 accuracy: 82.9 -> 83.3 | Helped |
| Replace MLP with MBConv (ablation) | tiny-MOAT-2 · ImageNet-1K, input 224² | top-1 accuracy: 79.8 -> 81.0 | Helped |
| Delegate downsampling to MBConv rather than average pooling, patch embedding, or strided depthwise convolution (ablation) | MOAT-0 · ImageNet-1K, input 224² | top-1 accuracy: 82.6; 82.8; 82.6 -> 83.3 | Helped |
| Use MBConv for downsampling and channel expansion in attention-first order (ablation) | MOAT-0 · ImageNet-1K, input 224² | top-1 accuracy: 82.9 -> 83.1 | Helped |
| Reverse attention and MBConv order while keeping the first attention layer in the same stage (ablation) | MOAT-0 · ImageNet-1K, input 224² | top-1 accuracy: 83.1 -> 83.3 | Helped |
| Add a 1×1 convolution to expand channels at the first attention layer (ablation) | Attention-first MBConv block · ImageNet-1K, MOAT-0, input 224² | top-1 accuracy: 83.2 vs 83.3 | No clear effect |
| Use MOAT blocks in stages 2–5 instead of MOAT blocks only in stages 4–5 (ablation) | MOAT-0 meta-architecture · ImageNet-1K, input 224² | top-1 accuracy: 83.3 -> 83.6 | Helped |
| Replace MOAT with MBConv in stage 4 (ablation) | MOAT-0 meta-architecture · ImageNet-1K, input 224² | top-1 accuracy: 83.3 -> 82.2 | Hurt |
| Replace MOAT blocks in every stage with MBConv (ablation) | MOAT-0 meta-architecture · ImageNet-1K, input 224² | top-1 accuracy: 83.3 -> 82.0 | Hurt |
| Increase input resolution from 224² to 384² | MOAT-0 · ImageNet-1K, 1K only | top-1 accuracy: 83.3 -> 84.6 | Helped |
Closest papers by meaning
- Mobile-Former: Bridging MobileNet and Transformer (CVPR 2022 · similarity 0.89)
- MobileODE: An Extra Lightweight Network (NeurIPS 2025 · similarity 0.88)
- IFORMER: INTEGRATING CONVNET AND TRANSFORMER FOR MOBILE APPLICATION (ICLR 2025 · similarity 0.88)
- Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers (ICML 2024 · similarity 0.85)
- AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation (NeurIPS 2024 · similarity 0.85)
- Revisiting the Integration of Convolution and Attention for Vision Backbone (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.