Research map / Vision-language and multimodal / Cross-modal learning
Multi-modal Vision Pre-training for Medical Image Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BrainMVP, a multi-modal MRI self-supervised pre-training framework combining cross-modal masked reconstruction, modality-wise learnable template distillation, and modality-aware contrastive learning, pre-trained on 16,022 brain mpMRI scans.
Problem
Self-supervised pre-training for medical imaging is largely uni-modal and fails to model inter-modal correlations in naturally grouped multi-parametric MRI data, and it struggles with missing or mismatched modalities.
Approach
Masks part of one MRI modality with patches from another modality of the same patient (cross-modal masked reconstruction), replaces masked regions with learnable per-modality templates distilled via performance matching along pre-training trajectories, and adds a contrastive loss aligning the two masked views, operating on single-channel uni-modal inputs to handle arbitrary modality counts.
Key results
- BrainMVP with UniFormer achieves 76.80% average Dice on BraTS2023-PED, versus 67.65% for the best general SSL MIM method, and 83.64% Dice on VSseg versus 79.31% for M3AE.
- On BraTS2018 classification, BrainMVP reaches 0.8596 accuracy versus 0.7895 for M3AE and GVSL.
- Label-efficiency experiments show BrainMVP with 40% of labels matches or exceeds fully labeled baselines on several tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BrainMVP pre-training instead of M3AE | UniFormer · BraTS2023-PED | Dice Score (%): 74.14 -> 76.80 | Helped |
| use BrainMVP pre-training instead of M3AE | UniFormer · BraTS-MET | Dice Score (%): 70.39 -> 73.67 | Helped |
| use BrainMVP pre-training instead of M3AE | UniFormer · ISLES22 | Dice Score (%): 86.32 -> 86.60 | Helped |
| use BrainMVP pre-training instead of MG | UniFormer · MRBrainS13 | Dice Score (%): 75.51 -> 80.27 | Helped |
| use BrainMVP pre-training instead of M3AE | UniFormer · VSseg | Dice Score (%): 79.31 -> 83.64 | Helped |
| use BrainMVP pre-training instead of M3AE | UniFormer · UPENN-GBM | Dice Score (%): 89.63 -> 90.01 | Helped |
| use BrainMVP pre-training instead of M3AE | UniFormer · BraTS2018 | ACC: 0.7895 -> 0.8596 | Helped |
| add cross-modal reconstruction (ablation) | BrainMVP · BraTS2023-PED | Dice Score (%): 72.52% to 75.16% | Helped |
| add modality-wise data distillation (ablation) | BrainMVP with cross-modal reconstruction · BraTS2023-PED | Dice Score (%): 75.16 -> 75.87 | Helped |
| add modality-aware contrastive learning (ablation) | BrainMVP with cross-modal reconstruction and modality-wise data distillation · BraTS2023-PED | Dice Score (%): 75.87% to 76.80% | Helped |
| add cross-modal reconstruction (ablation) | BrainMVP · BraTS2018 | AUC: 0.7719 to 0.8056 | Helped |
| add modality-wise data distillation (ablation) | BrainMVP with cross-modal reconstruction · BraTS2018 | AUC: 0.8056 -> 0.9032 | Helped |
Surprising findings
- Pretraining with only 20% labeled data reaches Dice 66.41% on BraTS2023-PED, above the best baselines' fully-supervised 59.50%.
- Modality-aware contrastive learning is only enabled at epoch 1000, after the templates converge.
Limitations
- Pre-training data is limited to brain MRI, so generalization to other body regions and modalities is untested.
- Some baseline results are re-implemented on the authors' pre-training data rather than taken from original papers.
- Some reported gains are modest (e.g., 0.28% Dice on some benchmarks) and results rely on single runs without reported variance.
Experimental setup
Datasets: BraTS2021 BraTS2023-SSA BraTS2023-MEN UCSF-PDGM IXI BraTS2023-PED BraTS2023-MET ISLES22 MRBrainS13 UPENN-GBM VSseg BraTS2018
Benchmarks: BraTS2023-PED BraTS2023-MET ISLES22 MRBrainS13 VSseg UPENN-GBM BraTS2018 ADNI ADHD-200 ABIDE-I
Metrics: Dice score HD95 accuracy AUC F1 score
Baselines: UNETR UNET3D UniFormer Swin-UNETR MAE3D SimMIM MoCov3 MG (Models Genesis) TransVW GVSL VoCo DAE
Builds on: M3AE dataset distillation (trajectory matching) Models Genesis VoCo UniFormer
Closest papers by meaning
- Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning (CVPR 2024 · similarity 0.88)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.87)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.87)
- Autoregressive Sequence Modeling for 3D Medical Image Representation (AAAI 2025 · similarity 0.86)
- Modality-Agnostic Structural Image Representation Learning for Deformable Multi-Modality Medical Image Registration (CVPR 2024 · similarity 0.86)
- Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.