Research map / Vision-language and multimodal / Cross-modal learning
Towards Unsupervised Multi-modal Semantic Segmentation
Haitian Zhang, Thai Nguyen, Xiangyuan Wang, Mohan Liu, Addison Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| naively fuse depth with RGB | STEGO · NYU-Depth-v2 (ViT-S/16) | mIoU: 28.8 -> 25.3 | Hurt |
| naively fuse depth with RGB | EAGLE · NYU-Depth-v2 (ViT-S/16) | mIoU: 27.4 -> 20.1 | Hurt |
| replace STEGO RGB-only with UniM2 depth fusion | STEGO · NYU-Depth-v2 (ViT-B/16) | mIoU: 31.7 -> 38.1 | Improved |
| replace STEGO RGB-only with UniM2 thermal fusion | STEGO · MFNet (ViT-B/16) | mIoU: 35.9 -> 45.7 | Improved |
| add CMCS and MSN to baseline (ablation) | UniM2 baseline · NYU-Depth-v2 (ViT-S/16) | mIoU: 25.0 -> 31.3 | Improved |
| add CMH (Cross-modal Harmonizer) (ablation) | UniM2 (CMCS + MSN) · NYU-Depth-v2 (ViT-S/16) | mIoU: 31.3 -> 36.9 | Improved |
| add MSN feature refinement before fusion (ablation) | UniM2 (CMCS + CMH) · NYU-Depth-v2 (ViT-S/16) | mIoU: 34.2 -> 36.9 | Improved |
| anchor on RGB only instead of no anchors (ablation) | UniM2 · NYU-Depth-v2 (ViT-S/16) | mIoU: 19.8 -> 36.9 | Improved |
| anchor on RGB only instead of Depth only (ablation) | UniM2 · NYU-Depth-v2 (ViT-S/16) | mIoU: 27.6 -> 36.9 | Improved |
| use Conv fusion instead of Sum fusion (ablation) | UniM2 · NYU-Depth-v2 (ViT-S/16) | mIoU: 33.0 -> 36.9 | Improved |
| use Conv fusion instead of Max fusion (ablation) | UniM2 · NYU-Depth-v2 (ViT-S/16) | mIoU: 30.5 -> 36.9 | Improved |
| use UniM2 instead of STEGO with four modalities | STEGO · MCubeS (ViT-S/16, INAD) | mIoU: 18.3 -> 20.7 | Improved |
Closest papers by meaning
- Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation (CVPR 2025 · similarity 0.91)
- OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation (NeurIPS 2025 · similarity 0.90)
- Integrating Low-Level Visual Cues for Enhanced Unsupervised Semantic Segmentation (AAAI 2025 · similarity 0.87)
- Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.87)
- UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior (NeurIPS 2024 · similarity 0.87)
- UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
