Research map / Vision-language and multimodal / Human and embodied modeling
CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CLIMB, a 4.51M-sample, 44-dataset multimodal clinical benchmark spanning 2D/3D imaging, time series, graphs and multimodal data, and shows multitask pretraining on it helps understudied tasks, few-shot transfer, and unimodal-to-multimodal fusion.
Problem
Existing clinical benchmarks and models cover only a small set of modalities and tasks (mostly images and text), which limits development of large-scale holistic multimodal clinical foundation models.
Approach
Unify 44 public clinical datasets across 15 modalities into one benchmark with a shared label vocabulary and a QA variant (CLIMB-QA), then use multitask pretraining on it per modality and evaluate encoders, few-shot transfer, and unimodal-to-multimodal fusion.
Key results
- Multitask pretraining improves ECG dataset Ga AUC from 0.474 to 0.704 (+23 points) versus single-task training; COVID-US gains 0.3254 AUC.
- CLIMB-pretrained ConvNeXtV2 reaches 0.877 accuracy on CLIMB-QA versus 0.555 for fine-tuned LLaVA-Med average.
- Pretrained encoders on CLIMB achieve 2.61 MAE on length-of-stay with cross-attention fusion versus 2.77 for SoTA encoders, and 15.9% higher 8-shot mortality AUC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multitask rather than single-task pretraining | Clinical models on CLIMB · COVID-US | AUC: AUC gain of 0.3254 | Helped |
| Use multitask rather than single-task pretraining | ECG models · Georgia | AUC: from 0.474 to 0.704 | Helped |
| Use CLIMB-pretrained rather than standard pretrained weights | Few-shot clinical models · Ultrasound | AUC: 29.1% | Helped |
| Use CLIMB-pretrained rather than standard pretrained weights | Few-shot clinical models · CT scans | AUC: 28.7% | Helped |
| Use CLIMB-pretrained rather than standard pretrained weights | ECG-JEPA · ECG | performance: 36.8% | Helped |
| Use ConvNeXTv2 rather than MedViT | Multitask visual encoder · Overall imaging | performance: 35.9% | Helped |
| Fine-tune on CLIMB-QA rather than evaluate zero-shot | LLaVa-Med · CLIMB-QA | accuracy: 28.7% percentage points | Helped |
| Use CLIMB-pretrained encoders rather than other pretrained encoders | Multimodal MIMIC-IV prediction models · LOS prediction | MAE: 2.77 vs 2.61 | Helped |
| Use MLP fusion rather than cross-attention fusion | MIMIC-IV multimodal model · 48 IHM full | AUC: 0.786 -> 0.961 | Helped |
| Use cross-attention rather than MLP fusion | MIMIC-IV multimodal model · LOS prediction | MAE: 2.84 -> 2.61 | Helped |
| Use CLIMB-pretrained rather than publicly pretrained encoders | MIMIC-IV multimodal model with MLP fusion · 48 IHM 8-shot | AUC: 0.556 -> 0.672 | Helped |
Surprising findings
- General-domain ConvNeXtV2 outperforms clinical-specific encoders such as MedViT by 35.9% in the visual domain.
- Late fusion achieves near-zero sensitivity on 48-hour mortality prediction, effectively predicting the majority class.
Limitations
- Many constituent datasets require credentialed access or data use agreements, so the full benchmark is not fully open.
- Clinical LVLMs (LLaVA-Med) lag far behind dedicated vision encoders on CLIMB-QA, and dataset-specific SoTA models still outperform the pretrained models in some cases.
Experimental setup
Datasets: CLIMB PTB-XL Chapman-Shaoxing Georgia CPSC IIIC TUAB TUEV MIMIC-CXR CheXpert VinDr-CXR MIMIC-IV
Benchmarks: CLIMB CLIMB-QA MIMIC-IV
Metrics: AUC sensitivity specificity accuracy MAE
Baselines: MedViT PMC-CLIP RAD-DINO SBB2 Swin Transformer EVA-2 InternViT LLaVA-Med UniTS BIOT
Builds on: ConvNeXt V2 ECG-JEPA BIOT UniTS LLaVA-Med
Closest papers by meaning
- QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training (NeurIPS 2025 · similarity 0.86)
- GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.86)
- CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations (NeurIPS 2026 · similarity 0.86)
- MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images (CVPR 2026 · similarity 0.86)
- Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation? (NeurIPS 2024 · similarity 0.85)
- MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.