Research map / Architectures and efficiency / CNN design
RepViT: Revisiting Mobile CNN From ViT Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Modernizes MobileNetV3-L step by step with lightweight-ViT design choices (RepViT block, macro and micro changes) to produce RepViT, a family of pure CNNs that beats lightweight ViTs on iPhone 12 latency and ImageNet accuracy, and a RepViT-SAM encoder.
Problem
Lightweight ViTs have recently beaten lightweight CNNs on accuracy and mobile latency, but the architectural differences between the two families (block structure, macro and micro design) have not been systematically examined.
Approach
Start from MobileNetV3-L and incrementally adopt lightweight-ViT designs (separate token/channel mixer with re-parameterized depthwise conv, narrower expansion ratio with wider channels, early-conv stem, deeper separate downsampling, simple classifier, ViT-like stage ratio, 3x3 kernels, cross-block SE placement), measuring on-device latency at each step to yield a pure CNN family.
Key results
- RepViT-M1.0 reaches 80.0% top-1 on ImageNet-1K at 1.0 ms latency on iPhone 12 (with distillation), and RepViT-M2.3 reaches 83.7% at 2.3 ms.
- Without distillation, RepViT-M2.3 gets 82.5% top-1 vs. 81.4% for PoolFormer-S36 at 3.5 ms latency, and RepViT-M1.0 gets 78.6% at 1.0 ms vs. 75.9% for MobileOne-S1 at 0.9 ms.
- RepViT-SAM is nearly 10x faster than MobileSAM on MacBook M1 Pro (44.8 ms vs. 482.2 ms encoder latency) and is the only one of the SAM variants that runs on iPhone 12 for 1024x1024 inputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| modernize MobileNetV3-L into RepViT | MobileNetV3-L · ImageNet-1K, 300 epochs, without distillation | Top-1 accuracy: 71.5 -> 77.4 | Helped |
| use a RepViT block that separates token and channel mixers with structural re-parameterization | MobileNetV3-L · ImageNet-1K, without distillation | Top-1 accuracy: 71.5 -> 68.3 | Hurt |
| reduce channel-mixer expansion ratio to 2 and increase network width | RepViT block · ImageNet-1K, without distillation | Top-1 accuracy: 68.3% -> 73.5% | Helped |
| replace the complex stem with two stride-2 3×3 early convolutions | RepViT · ImageNet-1K, without distillation | Top-1 accuracy: 73.5% -> 73.9% | Helped |
| use deeper, separate downsampling layers | RepViT · ImageNet-1K, without distillation | Top-1 accuracy: 73.9% -> 75.4% | Helped |
| replace the complex classifier with global average pooling and one linear layer | RepViT · ImageNet-1K, without distillation | Top-1 accuracy: 0.6% | Hurt |
| use stage ratio 1:1:7:1 and deepen the layout to 2:2:14:2 | RepViT · ImageNet-1K, without distillation | Top-1 accuracy: 75.4% -> 76.9% | Helped |
| replace larger convolution kernels with 3×3 convolutions | RepViT · ImageNet-1K | Top-1 accuracy: 76.9% -> 76.9% | No clear effect |
| use cross-block SE placement | RepViT · ImageNet-1K | Top-1 accuracy: 76.9% -> 77.4% | Helped |
| add structural re-parameterization multi-branch topology during training (ablation) | RepViT · ImageNet-1K | Top-1 accuracy: 78.47% -> 78.74% | Helped |
| add SE layers to the cross-block placement instead of removing all SE (ablation) | RepViT-M0.9 · ImageNet-1K | Top-1 accuracy: 77.92% -> 78.74% | Helped |
| use cross-block SE rather than SE in every block (ablation) | RepViT-M1.5 · ImageNet-1K | Latency: 1.58 ms -> 1.52 ms | Helped |
Surprising findings
- Simply reducing the expansion ratio and increasing width on the original MobileNetV3 block gave inferior accuracy (73.0%) compared with the RepViT block version (73.5%) at similar latency.
- A stage ratio of 1:1:7:1 and deeper layout raised accuracy while adding little latency, showing ViT-style depth distribution helps CNNs.
Limitations
- Latency is measured only on iPhone 12 with Core ML Tools; results on other hardware are not reported.
- Gelu activation was substituted into MobileNetV3-L to avoid unsupported ops, so the baseline differs from the original.
Experimental setup
Datasets: ImageNet-1K MS COCO 2017 ADE20K SA-1B BSDS500
Benchmarks: ImageNet-1K MS COCO 2017 ADE20K BSDS500 SegInW
Metrics: top-1 accuracy latency (ms) on iPhone 12 throughput APbox APmask mIoU ODS OIS AP
Baselines: MobileOne EfficientFormerV2 FastViT SwiftFormer MobileViG EfficientFormer PoolFormer MobileSAM ViT-B-SAM ViT-H-SAM
Builds on: MobileNetV3 MetaFormer RepVGG EfficientFormer SAM
Closest papers by meaning
- HydraViT: Stacking Heads for a Scalable ViT (NeurIPS 2024 · similarity 0.88)
- SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design (CVPR 2024 · similarity 0.88)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.88)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.87)
- Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers (ICML 2024 · similarity 0.87)
- Real-time Core-Periphery Guided ViT with Smart Data Layout Selection on Mobile Devices (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.