Research map / Vision-language and multimodal / CLIP and zero-shot
Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MFuser, a Mamba-based framework that jointly adapts a frozen VFM (DINOv2) and a frozen VLM (CLIP/EVA02-CLIP/SigLIP) for domain-generalized semantic segmentation, achieving 68.20 mIoU on G→{C,B,M} and 71.87 on C→{B,M}.
Problem
Domain generalized semantic segmentation relies on either VFMs or VLMs alone, which have complementary weaknesses (fine-grained locality vs. text alignment); naively combining them doubles patch sequence length, making attention-based fusion costly.
Approach
Use Mamba (state-space) blocks as parameter-efficient co-adapters that concatenate patch tokens from a frozen VFM and VLM at each layer (MVFuser, with parallel sequential SSM and spatial conv branches), and a hybrid attention-Mamba module (MTEnhancer) that refines class text embeddings with fused visual features to serve as Mask2Former queries.
Key results
- Achieves 68.20 mIoU averaged over G→{C,B,M} with DINOv2 + EVA02-CLIP, about 2.15 mIoU above prior state of the art on average.
- Achieves 71.87 mIoU averaged over C→{B,M}, improving the average by 1.43 mIoU.
- Naive concatenation of frozen VFM and VLM features yields 64.70 mIoU, below VFM-only (65.13) and VLM-only (66.15).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse DINOv2 and CLIP features with MFuser | DGSS segmentation · G→C | mIoU: 68.88 -> 71.24 | Helped |
| Fuse DINOv2 and EVA02-CLIP features with MFuser | DGSS segmentation · G→B | mIoU: 61.64 -> 63.13 | Helped |
| Fuse DINOv2 and SIGLIP features with MFuser | DGSS segmentation · G→M | mIoU: 70.10 -> 71.71 | Helped |
| Fuse DINOv2 and EVA02-CLIP features with MFuser | DGSS segmentation · C→B | mIoU: 64.72 -> 65.81 | Helped |
| Fuse DINOv2 and CLIP features with MFuser | DGSS segmentation · C→M | mIoU: 76.40 -> 78.10 | Helped |
| Replace MVFuser with concatenated self-attention (ablation) | MFuser on G→{C, B, M} · G→{C, B, M} | mIoU: 68.20 -> 67.89 | Hurt |
| Replace MVFuser with separate self-attention (ablation) | MFuser on G→{C, B, M} · G→{C, B, M} | mIoU: 68.20 -> 67.48 | Hurt |
| Replace MVFuser with bidirectional deformable attention (ablation) | MFuser on G→{C, B, M} · G→{C, B, M} | mIoU: 68.20 -> 66.91 | Hurt |
| Use both VFM and VLM visual features with shared MVFuser rather than only one encoder or unadapted concatenation (ablation) | MFuser on G→{C, B, M} · G→{C, B, M} | mIoU: 65.13; 66.15; 64.70 -> 68.20 | Helped |
| Replace MVFuser with convolution-based fusion (ablation) | MFuser on G→{C, B, M} · G→{C, B, M} | mIoU: 68.20 -> 66.83 | Hurt |
| Replace MVFuser with cross-attention fusion (ablation) | MFuser on G→{C, B, M} · G→{C, B, M} | mIoU: 68.20 -> 66.87 | Hurt |
| Use separate MVFusers instead of shared feature interaction (ablation) | MFuser on G→{C, B, M} · G→{C, B, M} | mIoU: 68.20 -> 67.68 | Hurt |
Surprising findings
- Naive concatenation of frozen VFM and VLM features performs worse than using either encoder alone (64.70 vs 65.13 and 66.15 mIoU).
Limitations
- Requires running two large frozen encoders (VFM and VLM) at inference, doubling the patch sequence length.
Experimental setup
Datasets: GTAV Cityscapes BDD100K Mapillary
Benchmarks: GTAV→Cityscapes GTAV→BDD100K GTAV→Mapillary Cityscapes→BDD100K Cityscapes→Mapillary
Metrics: mIoU
Baselines: Rein SET tqdm VLTSeg CLOUDS SHADE IBAFormer
Builds on: Mamba Rein tqdm Mask2Former
Closest papers by meaning
- Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation (NeurIPS 2025 · similarity 0.89)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.89)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.89)
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.88)
- FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation (CVPR 2025 · similarity 0.88)
- Mamba-Reg: Vision Mamba Also Needs Registers (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.