Research map / Detection and segmentation / Open-set and few-shot detection
UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniMRSeg, a single set of parameters for multi-modal segmentation under arbitrary missing-modality combinations, trained via a three-stage hierarchical self-supervised compensation (reconstruction, contrastive learning, adapter-based consistency fine-tuning).
Problem
Multi-modal segmentation degrades when modalities are missing or corrupted at inference; existing methods need per-combination models or modality matching, which raises deployment cost.
Approach
Hierarchically compensates representation gaps between complete and incomplete modalities at input (modality dropout, shuffle, spatial masking with reconstruction), feature (NT-Xent contrastive learning across modality-complete/incomplete pairs), and output levels (hybrid feature/prediction consistency), using a frozen encoder plus a lightweight reverse attention adapter with a 3D Swin Transformer
Key results
- On BraTS2020 with 15 MRI combinations, UniMRSeg reports average Dice of 80.64 (Whole), 73.33 (Core), 63.10 (Enhancing), versus best prior PASSION at 76.39, 66.06, 55.90 respectively, with lowest std dev.
- On RGB-D salient object segmentation (STERE), average S-measure 0.893 vs 0.826 for best baseline PopNet; on RGB-T (VT1000) 0.899 vs 0.869 for SSLSOD.
- On SUN-RGBD, UniMRSeg average IoU 50.7 vs 48.3 for CMX, with std dev 3.1 vs 4.9 for CMX.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniMRSeg instead of the strongest listed baseline | BraTS2020 · BraTS2020 (average across modality combinations) | Whole Tumor Dice (%): 76.39 -> 80.64 | Helped |
| Use UniMRSeg instead of the strongest listed baseline | BraTS2020 · BraTS2020 (average across modality combinations) | Tumor Core Dice (%): 66.06 -> 73.33 | Helped |
| Use UniMRSeg instead of the strongest listed baseline | BraTS2020 · BraTS2020 (average across modality combinations) | Enhancing Tumor Dice (%): 58.53 -> 63.10 | Helped |
| Use UniMRSeg instead of PopNet | RGB-D salient object segmentation · STERE | S-measure: .826 -> .893 | Helped |
| Use UniMRSeg instead of the strongest listed baseline | RGB-T salient object segmentation · VT1000 | S-measure: .869 -> .899 | Helped |
| Use UniMRSeg instead of CMX | RGB-D semantic segmentation · SUN-RGBD | IoU: 48.3 -> 50.7 | Helped |
| Use UniMRSeg instead of the strongest listed baseline | BraTS2020 · BraTS2020 (PASSION split, average across modality combinations) | Whole Tumor Dice (%): 83.67 -> 85.97 | Helped |
| Use UniMRSeg instead of the strongest listed baseline | BraTS2020 · BraTS2020 (PASSION split, average across modality combinations) | Tumor Core Dice (%): 75.02 -> 77.12 | Helped |
| Use UniMRSeg instead of the strongest listed baseline | BraTS2020 · BraTS2020 (PASSION split, average across modality combinations) | Enhancing Tumor Dice (%): 55.55 -> 61.65 | Helped |
| Use UniMRSeg instead of the strongest listed baseline | BraTS2018 · BraTS2018 (average across modality combinations) | Whole Tumor Dice (%): 86.32 -> 88.90 | Helped |
| Use UniMRSeg instead of the strongest listed baseline | BraTS2018 · BraTS2018 (average across modality combinations) | Tumor Core Dice (%): 78.28 -> 80.67 | Helped |
| Use UniMRSeg instead of the strongest listed baseline | BraTS2018 · BraTS2018 (average across modality combinations) | Enhancing Tumor Dice (%): 63.84 -> 67.02 | Helped |
Surprising findings
- Single-stage joint training of all losses failed to converge and was highly unstable, motivating the staged design.
- Fine-tuning the encoder in stage 3 caused over 6.4% performance drop, so the encoder is frozen in that stage.
Limitations
- Three-stage training pipeline adds training complexity and overhead.
- Missing modalities are simulated by zero-filling (MRI) or copying existing modalities (RGB-D/T), which may not reflect real corruption.
- Some baselines were retrained or taken from other papers under different splits, making exact comparison infeasible.
Experimental setup
Datasets: BraTS2020 NJUD NLPR STERE VT5000 VT1000 SUN-RGBD BraTS2018
Benchmarks: BraTS2020 STERE VT1000 SUN-RGBD BraTS2018
Metrics: Dice S-measure IoU standard deviation across modality combinations average performance
Baselines: NestedFormer SFusion ShaSpec PASSION SSLSOD CAVER PopNet GateNet LSNet CONTRINET TokenFusion CMX
Builds on: masked autoencoders NT-Xent contrastive loss 3D U-Net Swin Transformer PASSION
Closest papers by meaning
- Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation (CVPR 2025 · similarity 0.88)
- Semantic-guided Masked Mutual Learning for Multi-modal Brain Tumor Segmentation with Arbitrary Missing Modalities (AAAI 2025 · similarity 0.86)
- OGP-Net: Optical Guidance Meets Pixel-Level Contrastive Distillation for Robust Multi-Modal and Missing Modality Segmentation (AAAI 2025 · similarity 0.86)
- OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation (NeurIPS 2025 · similarity 0.86)
- KMD: Koopman Multi-modality Decomposition for Generalized Brain Tumor Segmentation under Incomplete Modalities (CVPR 2025 · similarity 0.86)
- Modality-agnostic Domain Generalizable Medical Image Segmentation by Multi-Frequency in Multi-Scale Attention (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.