Research map / Vision-language and multimodal / Cross-modal learning
Incomplete Multi-modal Brain Tumor Segmentation via Learnable Sorting State Space Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LS3M, a Mamba-based segmentation network with learnable differentiable sorting (Sinkhorn + Gumbel-Softmax permutation matrices) and a global input strategy for brain tumor segmentation under missing MRI modalities, evaluated on BraTS2018/2020.
Problem
Brain tumor segmentation from incomplete multi-modal MRI, where some modalities (e.g., T1ce) are missing; existing Mamba scanning strategies fail to preserve 3D spatial structure and long-range semantic correlations.
Approach
Replaces non-differentiable token sorting in Mamba with a learnable, differentiable permutation matrix (Sinkhorn over Gumbel-Softmax rows) that reorders 3D MRI tokens per patient, scales it to larger resolutions by bilinear interpolation, fuses modality-specific matrices with a missing-modality code, and uses a series bi-directional SSM (S3M) block plus a global input strategy with a relative posi
Key results
- Average DSC improvements of 1.22%, 1.31%, and 2.3% over the second-ranked method for WT, TC, and ET on BraTS2020 (Table 1).
- Ablation on BraTS2020: full LS3M reaches WT 88.22, TC 79.76, ET 63.60 DSC, vs baseline 85.43, 76.17, 58.02.
- Learnable sorting (SortP) adds gain over GIS+S3M (WT 87.17 to 88.22, ET 62.94 to 63.60).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LS3M instead of GSS | BraTS2020 brain tumor segmentation · Average across modality settings, WT | Dice (%): 87.00 -> 88.22 | Helped |
| Use LS3M instead of GSS | BraTS2020 brain tumor segmentation · Average across modality settings, TC | Dice (%): 78.45 -> 79.76 | Helped |
| Use LS3M instead of M3AE | BraTS2020 brain tumor segmentation · Average across modality settings, ET | Dice (%): 61.30 -> 63.60 | Helped |
| Use learnable sorted scanning instead of no specific scanning strategy (ablation) | LS3M · BraTS2020, WT | Dice: 85.43 -> 86.94 | Helped |
| Use learnable sorted scanning instead of no specific scanning strategy (ablation) | LS3M · BraTS2020, TC | Dice: 76.17 -> 78.01 | Helped |
| Use learnable sorted scanning instead of no specific scanning strategy (ablation) | LS3M · BraTS2020, ET | Dice: 58.02 -> 62.06 | Helped |
| Add global input strategy without relative position mask (ablation) | LS3M baseline · BraTS2020, WT | Dice: 85.43 -> 85.99 | Helped |
| Add a relative position mask to the global input strategy (ablation) | LS3M · BraTS2020, WT | Dice: 85.99 -> 86.23 | Helped |
| Add S3M to the global input strategy (ablation) | LS3M · BraTS2020, WT | Dice: 86.23 -> 87.17 | Helped |
| Add learnable sorting to GIS + S3M (ablation) | LS3M · BraTS2020, WT | Dice: 87.17 -> 88.22 | Helped |
| Use unidirectional scanning instead of no specific scanning strategy (ablation) | LS3M · BraTS2020, WT | Dice: 85.43 -> 85.20 | Hurt |
| Use random scanning instead of no specific scanning strategy (ablation) | LS3M · BraTS2020, WT | Dice: 85.43 -> 84.23 | Hurt |
Limitations
- Relies on simulated missing modalities by zeroing feature maps rather than real clinical missingness.
- Evaluated only on BraTS2018 and BraTS2020 datasets.
- Permutation matrix computation is quadratic in sequence length (L x L), which limits scaling to larger volumes.
Experimental setup
Datasets: BraTS2018 BraTS2020
Benchmarks: BraTS2018 BraTS2020
Metrics: Dice coefficient 95% Hausdorff distance sensitivity specificity
Baselines: HeMiS U-HVED RFNet UNet-MFI mmFormer MMANet ShaSpec M3AE GSS
Builds on: Mamba Vim SS2D LocalMamba Gumbel-Softmax Sinkhorn operator
Closest papers by meaning
- Semantic-guided Masked Mutual Learning for Multi-modal Brain Tumor Segmentation with Arbitrary Missing Modalities (AAAI 2025 · similarity 0.88)
- KMD: Koopman Multi-modality Decomposition for Generalized Brain Tumor Segmentation under Incomplete Modalities (CVPR 2025 · similarity 0.87)
- TMFormer: Token Merging Transformer for Brain Tumor Segmentation with Missing Modalities (AAAI 2024 · similarity 0.86)
- Federated Modality-Specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation (AAAI 2024 · similarity 0.85)
- UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation (NeurIPS 2025 · similarity 0.85)
- NaMa: Neighbor-Aware Multi-Modal Adaptive Learning for Prostate Tumor Segmentation on Anisotropic MR Images (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.