Research map / Architectures and efficiency / Attention and transformers
Efficient Modulation for Vision Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EfficientMod, a convolution-based efficient modulation block (simplified VAN/FocalNet modulation with fused MLP and a single large-kernel depthwise context branch) for fast efficient vision networks, optionally hybridized with self-attention.
Problem
Modulation-based vision networks (VAN, FocalNet) are theoretically efficient but slow at inference on limited hardware due to fragmented operations and redundant projections; efficient networks need better accuracy-latency trade-offs on GPU and CPU.
Approach
Abstracts the modulation mechanism as p(ctx(x) ⊙ v(x)) and fuses the MLP into the modulation design, with a simplified context branch using one 7x7 depthwise convolution in a channel-reduced setting, yielding a MBConv-like block that is faster than MBConv at similar params/FLOPs.
Key results
- EfficientMod-s reaches 81.0% top-1 on ImageNet-1K at 5.5 ms GPU latency (P100), 0.6 above EfficientFormerV2-S2 (80.4%) with about 25% lower GPU latency (7.3 ms).
- EfficientMod-s hybrid achieves 46.0 mIoU on ADE20K with Semantic FPN, 3.6 above EfficientFormerV2-s2 (42.4).
- On COCO with Mask R-CNN 1x, EfficientMod-s reaches 43.6 box AP versus 43.4 for EfficientFormerV2-s2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EfficientMod-s instead of EfficientFormerV2-S2 | ImageNet-1K classification · ImageNet-1K | top-1 accuracy (%): 80.4 -> 81.0 | Helped |
| use EfficientMod-s instead of EfficientFormerV2-S2 | ImageNet-1K classification · ImageNet-1K | GPU latency: 7.3 -> 5.5 ms; 25% | Helped |
| use hybrid EfficientMod-s with attention blocks instead of convolution-only EfficientMod-s (ablation) | EfficientMod-s · ImageNet-1K | top-1 accuracy (%): 80.5 -> 81.0 | Helped |
| add knowledge distillation (ablation) | EfficientMod-s · ImageNet-1K | top-1 accuracy (%): 81.0 -> 81.9 | Helped |
| replace element-wise multiplication with summation (ablation) | EfficientMod-s convolutional model · ImageNet-1K | top-1 accuracy (%): 1% | Hurt |
| use Repeat instead of Reshape (ablation) | EfficientMod-s · GPU latency | latency: 5.1% | Helped |
| use Repeat instead of Reshape (ablation) | EfficientMod-s · CPU latency | latency: 21.8% | Hurt |
| remove Layer Normalization and add Batch Normalization after each convolution (ablation) | EfficientMod-xxs (Conv) · iPhone 13 ImageNet-1K | latency: 1.2ms -> 0.9 ms | Helped |
| remove Layer Normalization and add Batch Normalization after each convolution (ablation) | EfficientMod-xxs (Conv) · iPhone 13 ImageNet-1K | top-1 accuracy (%): 75.3% -> 74.7% | Hurt |
| adjust channel and block numbers (ablation) | EfficientMod-xxs (Conv) with no LN and BN · iPhone 13 ImageNet-1K | top-1 accuracy (%): 74.7% -> 75.2% | Helped |
| use hybrid EfficientMod-s instead of convolution-only EfficientMod-s (ablation) | ADE20K semantic segmentation · ADE20K | mIoU: 43.5 -> 46.0 | Helped |
| use hybrid EfficientMod-s instead of EfficientFormerV2-s2 | ADE20K semantic segmentation · ADE20K | mIoU: 42.4 -> 46.0; 3.6 | Helped |
Surprising findings
- A100 showed higher latency than V100 at batch size 1 for almost all models, attributed to low GPU utilization.
- The paper reports that replacing element-wise multiplication with summation drops top-1 by about 1%, showing the modulation operation itself matters.
Limitations
- Scalability of efficient designs is understudied; large latency gap remains across scales.
- Large kernels or attention blocks may not be the most efficient way to enlarge receptive fields.
- Simplified context branch may slightly degrade performance as a compromise for efficiency.
Experimental setup
Datasets: ImageNet-1K MS COCO ADE20K
Benchmarks: ImageNet-1K MS COCO ADE20K
Metrics: top-1 accuracy GPU latency CPU latency AP box AP mask mIoU params FLOPs
Baselines: MobileNetV2 FasterNet EdgeViT MobileOne MobileViT EfficientFormerV2 MobileViTv2 PoolFormer VAN-B0 FocalNet
Builds on: VAN FocalNet MobileNetV2 MetaFormer ShuffleNetV2
Closest papers by meaning
- EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality (CVPR 2025 · similarity 0.85)
- FreeNet: Liberating Depth-Wise Separable Operations for Building Faster Mobile Vision Architectures (AAAI 2025 · similarity 0.85)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.84)
- GSPN-2: Efficient Parallel Sequence Modeling (NeurIPS 2025 · similarity 0.83)
- SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design (CVPR 2024 · similarity 0.83)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.