Research map / Detection and segmentation / CNN features
IFORMER: INTEGRATING CONVNET AND TRANSFORMER FOR MOBILE APPLICATION
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes iFormer, a mobile hybrid network that uses a streamlined ConvNeXt for early stages and a single-head modulation attention (SHMA) for later stages, optimized for on-device iPhone 13 latency.
Problem
Designing lightweight vision networks that balance accuracy and real on-device latency on mobile hardware, where multi-head attention's reshaping operations are memory-expensive and FLOPs/params correlate poorly with latency.
Approach
Progressively lighten ConvNeXt for latency on an iPhone 13 (BN instead of LN, overlapping stem, stage-ratio rebalancing), then replace later-stage multi-head attention with a single-head modulation attention whose context branch is multiplied element-wise with a sigmoid-gated feature branch, plus conditional positional encodings.
Key results
- iFormer-M reaches 80.4% ImageNet-1K Top-1 at 1.10 ms on iPhone 13, 0.5% above MobileNetV4-Conv-M (79.9%).
- iFormer-M with distillation reaches 81.1% Top-1 at 1.10 ms, versus 80.7% for RepViT-M1.1*.
- iFormer-L reaches 42.2 AP box on COCO Mask R-CNN and 44.5 mIoU on ADE20K at 6.60 ms backbone latency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the non-overlapping patchify stem with overlapping convolutions (ablation) | Lightweight ConvNeXt baseline · ImageNet-1K | Top-1 accuracy: 74.9% -> 76.7% | Helped |
| Replace LN with BN and non-overlapping downsampling with overlapping downsampling (ablation) | Overlapping-stem model · ImageNet-1K | Top-1 accuracy: 77.10% | Helped |
| Increase stage depths from (2,2,6,2) to (3,3,9,3) (ablation) | Lightweight ConvNeXt baseline · ImageNet-1K | Top-1 accuracy: 77.1% to 78.8% | Helped |
| Change stage depths from (3,3,9,3) to (2,2,18,2) (ablation) | Deeper ConvNeXt · ImageNet-1K | Top-1 accuracy: 79.1% | Helped |
| Use a 7×7 rather than 3×3 convolution kernel in the basic block (ablation) | Convolutional baseline · ImageNet-1K | latency: 1.00 -> 1.01 | No clear effect |
| Replace MHA with single-head self-attention (ablation) | MHA baseline · ImageNet-1K | latency: 1.25× acceleration | Helped |
| Replace MHA with single-head self-attention (ablation) | MHA baseline · ImageNet-1K | Top-1 accuracy: 0.1% | Hurt |
| Use sigmoid activation and Post-BN rather than SiLU and Pre-LN in SHMA (ablation) | SHMA · ImageNet-1K | Top-1 accuracy: 80.3 -> 80.4 | Helped |
| Replace sigmoid activation with SiLU while retaining Post-BN (ablation) | SHMA · ImageNet-1K | training stability | Hurt |
| Replace all Stage 3 convolutional blocks with SHMA (ablation) | iFormer hybrid block configuration · ImageNet-1K | Top-1 accuracy: 79.4 -> 79.0 | Hurt |
| Remove a block in the final stage (ablation) | iFormer · ImageNet-1K | Top-1 accuracy: -0.7% | Hurt |
| Reduce all FFN expansion widths (ablation) | iFormer · ImageNet-1K | Top-1 accuracy: -0.6% | Hurt |
Surprising findings
- Replacing SiLU with sigmoid in the modulation branch is needed; SiLU training diverged when using Post-BN.
- Enlarging the stem kernel to 5x5 and using larger depthwise kernels had almost no latency cost on mobile.
Limitations
- Inference bottleneck for high-resolution inputs remains; window attention on 512x512 still costs 4-11 ms.
- Hyperparameters such as learning rate are not tuned per variant.
Experimental setup
Datasets: ImageNet-1K COCO 2017 ADE20K
Benchmarks: ImageNet-1K COCO 2017 ADE20K
Metrics: Top-1 accuracy iPhone 13 latency (ms) AP box AP mask mIoU
Baselines: MobileNetV4 FastViT RepViT EfficientFormerV2 SwiftFormer EfficientMod SHViT MobileOne SBCFormer ConvNeXt
Builds on: ConvNeXt MobileNetV4 Conditional positional encodings Swin Transformer DeiT
Closest papers by meaning
- DECO: Unleashing the Potential of ConvNets for Query-based Detection and Segmentation (ICLR 2025 · similarity 0.85)
- Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers (ICML 2024 · similarity 0.85)
- RepViT: Revisiting Mobile CNN From ViT Perspective (CVPR 2024 · similarity 0.85)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.84)
- Revisiting the Integration of Convolution and Attention for Vision Backbone (NeurIPS 2024 · similarity 0.84)
- MobileMamba: Lightweight Multi-Receptive Visual Mamba Network (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.