Research map / Detection and segmentation / CNN features
MaskViM: Domain Generalized Semantic Segmentation with State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaskViM, a VMamba-style SSM encoder-decoder with a learnable Gumbel-Softmax binary token mask and mask loss to suppress non-causal tokens for domain generalized semantic segmentation.
Problem
Domain generalized semantic segmentation (DGSS) with state space models (Mamba) underperforms because unfolding 2-D images into 1-D sequences creates non-causal tokens and source-domain history dominates target-domain reasoning.
Approach
A learnable binary token mask, trained via Gumbel-Softmax and regularized by a mask loss controlling mask ratio and position, is inserted into the VSS blocks of an SSM encoder-decoder to filter out non-causal tokens.
Key results
- MaskViM-T averages 53.0% mIoU in Cityscapes-to-others, versus 51.3% for HGFormer (Swin-T).
- In Cityscapes-to-Cityscapes-c, MaskViM-T averages 45.8% mIoU, +1.9 over HGFormer-T.
- Removing the mask loss drops average mIoU from 53.0% (both losses) to 36.8% with cross-entropy only; the mask ratio and position losses each recover part of this gap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learnable mask blocks in the first three encoder stages (ablation) | VMamba+linear · BDD validation (trained on Cityscapes) | mIoU: 46.7 -> 48.1 | Helped |
| Add mask blocks in all encoder stages, including the final stage (ablation) | VMamba+linear · BDD validation (trained on Cityscapes) | mIoU: 46.7 -> 45.1 | Hurt |
| Add mask to the decoder (ablation) | MaskViM · BDD validation (trained on Cityscapes) | mIoU: 50.5 -> 53.0 | Helped |
| Retain multi-scale aggregation instead of replacing it with a VSS block (ablation) | MaskViM without decoder mask · BDD validation (trained on Cityscapes) | mIoU: 49.2 -> 50.5 | Helped |
| Use mask-ratio and mask-position losses together (ablation) | MaskViM · BDD validation (trained on Cityscapes) | mIoU: 36.8 -> 53.0 | Helped |
| Add mask-position loss to mask-ratio loss (ablation) | MaskViM · BDD validation (trained on Cityscapes) | mIoU: 45.6 -> 53.0 | Helped |
| Add mask-ratio loss to mask-position loss (ablation) | MaskViM · BDD validation (trained on Cityscapes) | mIoU: 45.2 -> 53.0 | Helped |
| Set a uniform mask ratio λ = 0.5 across encoder stages (ablation) | MaskViM encoder · BDD validation (trained on Cityscapes) | mIoU: 49.8 -> 53.0 | Helped |
| Use MaskD depths (1,1,1) instead of (2,2,2) (ablation) | MaskViM decoder · BDD validation (trained on Cityscapes) | mIoU: 52.0 -> 53.0 | Helped |
| Set mask-loss weights α1 and α2 to (1.0, 1.0) (ablation) | MaskViM · BDD validation (trained on Cityscapes) | mIoU: 49.6 -> 53.0 | Helped |
| Use MaskViM instead of HGFormer-T | DGSS models · Cityscapes-to-Cityscapes-c | mIoU: 43.9 -> 45.8 | Helped |
| Use MaskViM instead of HGFormer with Swin-T | DGSS models · Cityscapes-to-others | mIoU: 51.3 -> 53.0 | Helped |
Surprising findings
- Applying the mask in the final (lowest-resolution, most semantic) encoder stage hurts performance (46.7% to 45.1%), so masking is restricted to earlier stages.
Limitations
- Mask ratio and position hyperparameters (lambda, alpha1, alpha2, depths) were tuned on the evaluation settings.
- Hardware and training cost details are not stated.
Experimental setup
Datasets: Cityscapes Mapillary GTAV SYNTHIA BDD Cityscapes-C
Benchmarks: Cityscapes-C BDD Mapillary GTAV SYNTHIA Cityscapes
Metrics: mIoU
Baselines: HGFormer Mask2former BlindNet SAN-SAW WildNet DAFormer HRDA UperNet VMamba
Builds on: VMamba Mamba Gumbel-Softmax HGFormer Swin Transformer
Closest papers by meaning
- Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation (NeurIPS 2025 · similarity 0.89)
- Collaborating Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.89)
- START: A Generalized State Space Model with Saliency-Driven Token-Aware Transformation (NeurIPS 2024 · similarity 0.88)
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.88)
- Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance (ICML 2025 · similarity 0.88)
- Exploring Semantic Consistency and Style Diversity for Domain Generalized Semantic Segmentation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.