Research map / Architectures and efficiency / Attention and transformers

EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality

CVPR 2025 · Architectures and efficiency › Attention and transformers · approach: Mamba architectures · 69 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EfficientViM, a lightweight Mamba-style vision backbone whose HSM-SSD layer moves channel mixing into compressed hidden states and adds multi-stage hidden state fusion, achieving better speed-accuracy trade-offs on ImageNet-1K and dense tasks.

Problem

Efficient global token mixing for resource-constrained vision models; the linear projections in SSD (state space duality) layers dominate runtime, making vision Mambas slower than lightweight CNN/ViT models.

Approach

Redesign the NC-SSD layer so gating and output channel-mixing operate on the reduced hidden state (N rows instead of L tokens), reducing cost from O(LD^2) to O(ND^2+LND); add single-head state-wise importance weights to remove memory-bound multi-head ops, and fuse logits from hidden states at every stage.

Key results

• EfficientViM-M2 reaches 75.4% top-1 at 300 epochs (75.8% at 450) with 17,005 im/s on RTX 3090, outperforming SHViT-S2 (75.2%, 15,899 im/s).
• EfficientViM-M4 reaches 79.6% top-1 (450 epochs) and 80.7% with distillation, versus SHViT-S4 at 79.4% and 80.2%.
• EfficientViM-M4 achieves 41.3 mIoU on ADE20K with SemanticFPN at 0.45 ms latency, versus EdgeViT-XXS at 39.7 mIoU.

Measured effects reported in the paper

Surprising findings

• Multi-head HSM-SSD is a bottleneck: memory-bound operations take almost a quarter of runtime, so the single-head variant was adopted with higher throughput and no accuracy loss.
• Parameter count is not a critical factor for memory and speed efficiency; models with many parameters (EfficientViT, SHViT, EfficientViM) show high throughput and low peak memory.

Limitations

• Parameter counts are larger than many prior lightweight models (e.g., 19.6M for M4), though memory use stays competitive.
• Gains over SHViT are small (often under 1%), and some comparisons rely on distillation or longer training schedules.

Experimental setup

Datasets: ImageNet-1K COCO-2017 ADE20K

Benchmarks: ImageNet-1K COCO-2017 ADE20K

Metrics: top-1 accuracy throughput (im/s) latency (ms) box AP mask AP mIoU peak memory parameters FLOPs

Baselines: SHViT EfficientViT FastViT MobileNetV3 EfficientFormerV2 EfficientVMamba MSVMamba ViM

Builds on: Mamba2 SSD VSSD non-causal SSD DeiT distillation Vision Mamba

Closest papers by meaning

• EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba (AAAI 2025 · similarity 0.89)
• MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.87)
• Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.87)
• Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.87)
• MobileMamba: Lightweight Multi-Receptive Visual Mamba Network (CVPR 2025 · similarity 0.87)
• Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (ICML 2024 · similarity 0.87)