Research map / Detection and segmentation / CNN features
SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SegMAN, a linear-time segmentation network combining a hybrid encoder (sliding-window neighborhood attention plus SS2D state space blocks) with a Mamba-based multi-scale decoder (MMSCopE) that scales with input resolution.
Problem
Semantic segmentation needs global context modeling, fine-grained local detail encoding, and multi-scale feature extraction at varying input resolutions, but prior methods (window attention, spatial-reduction attention, pooling-based decoders) lack one or more of these at high resolution or lose det
Approach
LASS token mixer serially stacks Natten sliding-window local attention and VMamba's SS2D for global context in each encoder block, and the MMSCopE decoder concatenates multi-resolution regionally aggregated feature maps (via pixel unshuffle) into a single SS2D scan to extract multi-scale context with full coverage.
Key results
- SegMAN-B reaches 52.6% mIoU on ADE20K (+1.6% over SegNeXt-L with 15% fewer GFLOPs) and 83.8% mIoU on Cityscapes.
- SegMAN-B Encoder achieves 85.1% ImageNet-1K top-1 accuracy, +1.5% over VMamba-S with fewer parameters.
- SegMAN-T achieves 43.0% ADE20K mIoU and roughly 3x the inference speed of EDAFormer-T on Cityscapes (34.9 vs 12.7 FPS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SegMAN-B instead of SegNeXt-L | semantic segmentation model · ADE20K | mIoU: 51.0 -> 52.6 | Helped |
| use SegMAN-B instead of SegFormer-B3 | semantic segmentation model · Cityscapes | mIoU: 81.7 -> 83.8 | Helped |
| use SegMAN-B instead of VWFormer-B3 | semantic segmentation model · COCO-Stuff-164K | mIoU: 46.8 -> 48.4 | Helped |
| use SegMAN-B Encoder instead of VMamba-S | ImageNet-1K classification backbone · ImageNet-1K | Top-1 accuracy: 83.6 -> 85.1 | Helped |
| replace LASS with MaxViT (ablation) | SegMAN-S Encoder · ADE20K | mIoU: 51.3 -> 47.2 | Hurt |
| arrange Natten and SS2D in parallel (ablation) | SegMAN-S Encoder · ADE20K | mIoU: 51.3 -> 48.9 | Hurt |
| remove the SS2D residual connection (ablation) | SegMAN-S Encoder · ADE20K | mIoU: 51.3 -> 50.2 | Hurt |
| replace SS2D with SRA (ablation) | SegMAN-S Encoder · ADE20K | mIoU: 51.3 -> 50.6 | Hurt |
| remove SS2D (ablation) | SegMAN-S Encoder · ADE20K | mIoU: 51.3 -> 47.4 | Hurt |
| replace Natten with convolution (ablation) | SegMAN-S Encoder · ADE20K | mIoU: 51.3 -> 49.5 | Hurt |
| remove Natten (ablation) | SegMAN-S Encoder · ADE20K | mIoU: 51.3 -> 49.8 | Hurt |
| remove the 64×64 feature map from MMSCopE (ablation) | SegMAN-S · ADE20K | mIoU: 51.3 -> 50.6 | Hurt |
Surprising findings
- Running independent Mamba scans per feature map instead of the single concatenated scan gives only +0.2 mIoU while dropping FPS from 128 to 65.
- Replacing SS2D in Stage 4 with global attention did not help; removing the SS2D residual connection gave only a small drop on classification (-0.3%).
Limitations
- Global attention is used in Stage 4 instead of SS2D, so the encoder is not purely linear-time at the top stage.
- Gains on ImageNet and segmentation are modest over strong baselines (e.g., ablation differences of 0.2-0.9 points).
Experimental setup
Datasets: ImageNet-1K ADE20K Cityscapes COCO-Stuff-164K
Benchmarks: ImageNet-1K ADE20K Cityscapes COCO-Stuff-164K
Metrics: top-1 accuracy mIoU GFLOPs FPS parameters
Baselines: SegFormer SegNeXt VWFormer EDAFormer CGRSeg Swin UperNet ViT-CoMer Mask2Former VMamba SparX-Mamba InternImage ConvNeXt
Builds on: VMamba Neighborhood Attention SegFormer Mamba
Closest papers by meaning
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.87)
- MaSS13K: A Matting-level Semantic Segmentation Benchmark (CVPR 2025 · similarity 0.86)
- Multi-Scale Representations by Varying Window Attention for Semantic Segmentation (ICLR 2024 · similarity 0.86)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.86)
- EOV-Seg: Efficient Open-Vocabulary Panoptic Segmentation (AAAI 2025 · similarity 0.85)
- Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.