Research map / Vision-language and multimodal / Cross-modal learning
TMFormer: Token Merging Transformer for Brain Tumor Segmentation with Missing Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TMFormer, a transformer that treats available MRI modality combinations as variable-length token sequences, merging tokens (UMB and MMB) instead of using zero-map substitutes for missing modalities in brain tumor segmentation.
Problem
Brain tumor segmentation from multi-modal MRI when one or more modalities are missing; existing catch-all methods fill missing modalities with zero maps, causing feature bias and redundant computation.
Approach
Treat each available-modality combination as a variable-length token sequence; adaptively merge spatially redundant uni-modal tokens guided by a coarse segmentation map (UMB) and merge inter-modal tokens ordered by modality contribution (MMB), with no zero-map substitutes and no computation on absent branches.
Key results
- On BraTS 2020, TMFormer improves average DSC over the second-ranked method by 0.5% (WT), 1.1% (TC), and 2.5% (ET).
- Ablation on BraTS 2020: the adaptive SRW+SAR uni-modal merging raises ET DSC from 62.20 to 63.43 over random-window merging without the coarse-map guidance.
- GFLOPs for TMFormer vary with the number of available modalities (e.g., 30.23 to 103.37 across 1-4 modalities on BraTS 2020 table), while baselines have fixed cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TMFormer rather than M3AE | Brain tumor segmentation · BraTS 2020 | WT DSC (%): 86.25 -> 87.27 | Helped |
| Use TMFormer rather than M3AE | Brain tumor segmentation · BraTS 2020 | TC DSC (%): 77.56 -> 78.66 | Helped |
| Use TMFormer rather than M3AE | Brain tumor segmentation · BraTS 2020 | ET DSC (%): 61.30 -> 63.43 | Helped |
| Replace UMB token merging with pooling (ablation) | TMFormer · BraTS 2020 | WT DSC (%): 82.92 -> 87.27 | Hurt |
| Replace UMB token merging with strided convolution (ablation) | TMFormer · BraTS 2020 | WT DSC (%): 82.21 -> 87.27 | Hurt |
| Use alternate token sampling in UMB (ablation) | TMFormer · BraTS 2020 | ET DSC (%): 48.00 -> 63.43 | Hurt |
| Use central token sampling in UMB (ablation) | TMFormer · BraTS 2020 | ET DSC (%): 49.43 -> 63.43 | Hurt |
| Use random token sampling without adaptive sampling in UMB (ablation) | TMFormer · BraTS 2020 | ET DSC (%): 62.20 -> 63.43 | Hurt |
| Use alternate token sampling in MMB (ablation) | TMFormer · BraTS 2020 | ET DSC (%): 62.89 -> 63.43 | Hurt |
| Reorder modality tokens in MMB (ablation) | TMFormer · BraTS 2020 | WT DSC (%): 86.98 -> 87.27 | Hurt |
| Use 1 stage instead of the full multi-scale design (ablation) | TMFormer · BraTS 2020 | ET DSC (%): 60.06 -> 63.43 | Hurt |
| Use 2 stages instead of the full multi-scale design (ablation) | TMFormer · BraTS 2020 | ET DSC (%): 62.53 -> 63.43 | Hurt |
Surprising findings
- Global token merging outperforms local merging within cubic windows in the ablation study.
Limitations
- Relies on a coarse segmentation map from the previous scale to guide token merging, adding dependency on the decoder's coarse prediction.
- Evaluated only on BraTS 2018 and 2020 with missing-modality settings derived from those datasets.
Experimental setup
Datasets: BraTS 2018 BraTS 2020
Benchmarks: BraTS 2018 BraTS 2020
Metrics: Dice Similarity Coefficient (DSC) parameters (M) GFLOPs
Baselines: HeMiS U-HVED RFNet UNet-MFI mmFormer M3AE
Builds on: Token Merging (ToMe) U-Net mmFormer RFNet
Closest papers by meaning
- DTMFormer: Dynamic Token Merging for Boosting Transformer-Based Medical Image Segmentation (AAAI 2024 · similarity 0.88)
- Incomplete Multi-modal Brain Tumor Segmentation via Learnable Sorting State Space Model (CVPR 2025 · similarity 0.86)
- KMD: Koopman Multi-modality Decomposition for Generalized Brain Tumor Segmentation under Incomplete Modalities (CVPR 2025 · similarity 0.85)
- Semantic-guided Masked Mutual Learning for Multi-modal Brain Tumor Segmentation with Arbitrary Missing Modalities (AAAI 2025 · similarity 0.85)
- Federated Modality-Specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation (AAAI 2024 · similarity 0.85)
- NaMa: Neighbor-Aware Multi-Modal Adaptive Learning for Prostate Tumor Segmentation on Anisotropic MR Images (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.