Research map / Vision-language and multimodal / Cross-modal learning
OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniSegmentor, a pretrain-and-finetune framework for multi-modal semantic segmentation built on a new synthetic five-modality ImageNet variant (ImageNeXt) with a pairwise RGB+one-modality pretraining scheme.
Problem
Multi-modal semantic segmentation typically initializes supplementary-modality branches from RGB-pretrained weights, causing encoding mismatch, and there is no large-scale pretraining data or flexible pretrain-and-finetune pipeline covering many visual modalities.
Approach
Build ImageNeXt by adding synthetic depth, thermal, LiDAR and event modalities to ImageNet-1K; pretrain a DFormer-based shared-encoder model on RGB plus one randomly selected modality per iteration; finetune with modality-specific stems/MLPs initialized from the pretrained weights and additive fusion across any number of supplementary modalities.
Key results
- OmniSegmentor (DFormer-L) reaches 57.6% mIoU on NYU Depthv2 and 69.2% on KITTI-360, about +4% over the prior best on KITTI-360.
- On EventScape RGB-D-E, OmniSegmentor with DFormer-L reaches 67.6% mIoU versus 65.0 for CMX (MiT-B4), and ImageNeXt pretraining beats RGB-only pretraining (e.g., 63.8 vs 59.7 with simple fusion).
- On MFNet (RGB-T), OmniSegmentor (DFormer-L) reaches 60.6% mIoU, versus 59.9 for CMNeXt (MiT-B4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ImageNeXt pretraining instead of RGB-only pretraining (ablation) | OmniSegmentor · MFNet (RGB-T) | mIoU: +2.4% | Helped |
| use ImageNeXt pretraining instead of RGB-only pretraining (ablation) | OmniSegmentor · KITTI-360 (RGB-L) | mIoU: 5.1% | Helped |
| pretrain with all five modalities rather than omit a supplementary modality (ablation) | ImageNeXt pretraining · NYU V2 (RGB-D) | mIoU: 54.3 -> 52.2 | Helped |
| add event to the pretraining modalities (ablation) | ImageNeXt pretraining · EventScape RGB-D-E | mIoU: 63.8 -> 62.9 | Helped |
| add thermal to the pretraining modalities (ablation) | ImageNeXt pretraining · MFNet (RGB-T) | mIoU: 56.4 -> 57.6 | Helped |
| use simple fusion instead of SQ-Hub (ablation) | RGB-only pretraining model · EventScape RGB-D-E | mIoU: 59.7 -> 61.0 | Hurt |
| use simple fusion instead of SQ-Hub (ablation) | ImageNeXt-pretrained OmniSegmentor · EventScape RGB-D-E | mIoU: 63.8 -> 63.7 | No clear effect |
| use separate MLPs instead of a shared MLP for supplementary modalities (ablation) | OmniSegmentor · EventScape RGB-D-E | mIoU: 66.7 -> 67.6 | Helped |
| add depth to pretraining modalities (ablation) | ImageNeXt pretraining · NYU V2 (RGB-D) | mIoU: 54.3 -> 54.4 | Helped |
| remove thermal from pretraining modalities (ablation) | ImageNeXt pretraining · MFNet (RGB-T) | mIoU: 57.6 -> 56.4 | Hurt |
Surprising findings
- Simultaneous pretraining on all modalities makes optimization difficult and drops ImageNet Top-1 accuracy relative to RGB-only pretraining.
- A sophisticated fusion module (SQ-Hub) gives no gain over simple addition fusion under ImageNeXt pretraining, though it helps under RGB-only pretraining.
Limitations
- Existing benchmarks cover only a subset of the five modalities and some rely on synthetic simulator data.
- Supplementary modalities in ImageNeXt are synthetically generated (depth, thermal, LiDAR, event), so real-sensor fidelity is not established.
Experimental setup
Datasets: ImageNet-1K N-ImageNet VT821 VT1000 VT5000 FLIR NYU Depthv2 SUNRGBD MFNet KITTI-360 EventScape DeLiVER
Benchmarks: NYU Depthv2 SUNRGBD MFNet KITTI-360 EventScape DeLiVER ImageNet-1K
Metrics: mIoU Top-1 accuracy parameters FLOPs FPS
Baselines: CMNeXt CMX DFormer HRFuser TokenFusion MultiMAE Omnivore
Builds on: DFormer CMNeXt CMX ImageNet
Closest papers by meaning
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.87)
- UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior (NeurIPS 2024 · similarity 0.87)
- Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation (CVPR 2025 · similarity 0.87)
- Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.86)
- Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation (AAAI 2024 · similarity 0.86)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.