Research map / Vision-language and multimodal / Human and embodied modeling
Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Med-ST pre-trains a medical vision-language model on MIMIC-CXR using frontal/lateral views (MoVE encoder), modality-weighted local alignment, and cross-modal bidirectional cycle consistency (FMC + RMR) over patient report sequences.
Problem
Medical vision-language pre-training typically uses only frontal image-report pairs, ignoring lateral views and the temporal ordering of a patient's historical studies.
Approach
Exploit spatial (frontal+lateral views via a Mixture of View Experts ViT) and temporal (patient study sequences) supervision already present in multimodal datasets, adding modality-weighted local alignment and a cross-modal cycle-consistency objective that combines forward classification with reverse Gaussian-prior regression.
Key results
- On MS-CXR-T 10-fold temporal image classification, Med-ST reaches 61.12% average accuracy vs. 59.02% for BioViL-T.
- On MS-CXR-T RadGraph sentence similarity, Med-ST reaches 83.76% accuracy and 84.60 AUROC vs. 78.81 and 81.39 for BioViL-T.
- On COVIDx with 100% training data Med-ST achieves 93.0% vs. 92.3% for MGCA; it underperforms at 1% data (71.2% vs. 74.8% for MGCA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Med-ST instead of BioViL-T | medical multimodal pre-training · MS-CXR-T 10-fold | Avg. Acc.: 59.02 -> 61.12 | Helped |
| Use Med-ST instead of BioViL-T | medical multimodal pre-training · MS-CXR-T RadGraph | accuracy: 78.81 -> 83.76 | Helped |
| Use Med-ST instead of BioViL-T | medical multimodal pre-training · MS-CXR-T RadGraph | AUROC: 81.39 -> 84.60 | Helped |
| Use Med-ST instead of strongest baseline | medical multimodal pre-training · RSNA test | accuracy: 67.25 -> 68.37 | Helped |
| Use Med-ST instead of strongest baseline | medical multimodal pre-training · RSNA test | F1: 54.87 -> 57.63 | Helped |
| Use Med-ST instead of strongest baseline | medical multimodal pre-training · RSNA test | AUROC: 75.27 -> 77.14 | Helped |
| Use Med-ST instead of MGCA | medical multimodal pre-training · COVIDx 1% training data | classification score: 74.8 -> 71.2 | Hurt |
| Use Med-ST instead of MGCA | medical multimodal pre-training · COVIDx 10% training data | classification score: 84.8 -> 87.7 | Helped |
| Use Med-ST instead of MGCA | medical multimodal pre-training · COVIDx 100% training data | classification score: 92.3 -> 93.0 | Helped |
| Use Med-ST instead of BioViL-T | medical multimodal pre-training · MS-CXR-T Swaps | accuracy: 94.29 -> 76.73 | Hurt |
| Add local alignment (ablation) | Med-ST · MS-CXR-T 10-fold | Avg. Acc.: 59.49 -> 61.12 | Helped |
| Add the temporal objective (ablation) | Med-ST · MS-CXR-T 10-fold | Avg. Acc.: 58.85 -> 61.12 | Helped |
Surprising findings
- On the Swaps sentence-similarity subset, Med-ST (76.73% accuracy) underperforms BioViL-T (94.29%), which the authors attribute to masked language modeling.
Limitations
- Underperforms at very low label fraction (1% COVIDx) and on the Swaps subset.
- Pre-training initialized from MGCA weights, so gains are partly relative to that initialization.
- Evaluation limited to a small set of downstream tasks on chest X-rays.
Experimental setup
Datasets: MIMIC-CXR
Benchmarks: MS-CXR-T RSNA pneumonia COVIDx
Metrics: accuracy F1 AUROC
Baselines: ConVIRT GLoRIA MGCA MedCLIP MRM BioViL BioViL-T MedKLIP
Builds on: CLIP MGCA BioViL-T ViT
Closest papers by meaning
- MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images (CVPR 2026 · similarity 0.89)
- BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (NeurIPS 2024 · similarity 0.88)
- MedSG-Bench: A Benchmark for Medical Image Sequences Grounding (NeurIPS 2025 · similarity 0.87)
- Autoregressive Sequence Modeling for 3D Medical Image Representation (AAAI 2025 · similarity 0.87)
- MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models (ICML 2026 · similarity 0.87)
- MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.