Research map / Vision-language and multimodal / Cross-modal learning
Semantic-guided Masked Mutual Learning for Multi-modal Brain Tumor Segmentation with Arbitrary Missing Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMML, a masked dual-branch mutual learning framework for brain tumor segmentation under arbitrary missing MRI modalities, adding hierarchical consistency constraints (pixel and feature level) and a training-only SAM-Med3D-guided refinement network.
Problem
Brain tumor segmentation from multi-modal MRI when arbitrary subsets of modalities are missing, which degrades performance and makes it hard to learn robust, discriminative multi-modal features.
Approach
Dual-branch student networks trained on randomly modality-masked inputs and supervised by a pixel-level bidirectional transfer (selecting reliable pixels by per-pixel CE) plus a feature-level inter-sample/inter-class relational distillation weighted by class uncertainty; each branch also has a training-only refinement UNet that fuses SAM-Med3D per-modality segmentation cues with initial prediction
Key results
- On BraTS 2018 across fifteen missing-modality combinations, SMML reaches a mean DSC of 76.7% versus 74.5% for the no-component baseline (mmFormer-based).
- SMML achieves average DSC of 75.1% on BraTS 2015 and 78.5% on BraTS 2020, outperforming the second-best methods by 1.3 and 1.7 points respectively.
- Ablation on BraTS 2018 shows the SRN alone adds 0.7 points, dual-branch alone adds 0.8 points, and the full model is 2.2 points above baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SMML instead of mmFormer | Brain tumor segmentation · BraTs 2015 | Mean DSC (%): 1.3% | Helped |
| Use SMML instead of SMU-Net | Brain tumor segmentation · BraTs 2020 | Mean DSC (%): 1.7% | Helped |
| Add the SRN refinement network (ablation) | mmFormer baseline · BraTs 2018 | Mean DSC (%): 0.7% | Helped |
| Add the dual-branch scheme (ablation) | mmFormer baseline · BraTs 2018 | Mean DSC (%): 0.8% | Helped |
| Add PBC (ablation) | Dual-branch mmFormer · BraTs 2018 | Mean DSC (%): 0.4% | Helped |
| Add FRC (ablation) | Dual-branch mmFormer · BraTs 2018 | Mean DSC (%): 0.5% | Helped |
| Add SRN (ablation) | Dual-branch mmFormer · BraTs 2018 | Mean DSC (%): 0.4% | Helped |
| Add dual-branch learning, PBC, FRC, and SRN (ablation) | mmFormer baseline · BraTs 2018 | Mean DSC (%): 2.2% | Helped |
Limitations
- Relies on SAM-Med3D per-modality segmentation cues, which are generated per modality and zeroed for missing ones.
- Evaluated only on three brain tumor MRI datasets; generalization to natural images is only in supplementary material.
- Improvements over the strongest baselines are modest (around 1.3 to 1.7 mean DSC points).
Experimental setup
Datasets: BraTS 2015 BraTS 2018 BraTS 2020
Benchmarks: BraTS 2015 BraTS 2018 BraTS 2020
Metrics: Dice Similarity Coefficient
Baselines: MCTSeg M3AE U-Net-MFI RFNet mmFormer SMU-Net
Builds on: mmFormer Segment Anything Model SAM-Med3D deep mutual learning
Closest papers by meaning
- Incomplete Multi-modal Brain Tumor Segmentation via Learnable Sorting State Space Model (CVPR 2025 · similarity 0.88)
- UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation (NeurIPS 2025 · similarity 0.86)
- KMD: Koopman Multi-modality Decomposition for Generalized Brain Tumor Segmentation under Incomplete Modalities (CVPR 2025 · similarity 0.86)
- TMFormer: Token Merging Transformer for Brain Tumor Segmentation with Missing Modalities (AAAI 2024 · similarity 0.85)
- Federated Modality-Specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation (AAAI 2024 · similarity 0.85)
- NaMa: Neighbor-Aware Multi-Modal Adaptive Learning for Prostate Tumor Segmentation on Anisotropic MR Images (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.