Research map / Vision-language and multimodal / Cross-modal learning
Multi-modal Medical Diagnosis via Large-small Model Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaCoMed, which fuses features from off-the-shelf single-modal medical large models via a mixture-of-modality-experts module and co-learns with a pretrained multi-modal small model through contrastive alignment, shared task head, and adaptive logit weighting.
Problem
Multi-modal medical diagnosis lacks large models because modal-complete paired medical data is scarce, while existing small multi-modal models have limited feature extraction; naive fusion of single-modal large models suffers from modality domination.
Approach
Combine a mixture-of-modality-experts fusion of frozen-style single-modal large model embeddings with a pretrained small multi-modal model, aligned by a contrastive loss with a shared task head, and dynamically weight their logits via one of four adaptive strategies driven by the similarity score.
Key results
- On MIMIC-IV-MM mortality, AdaCoMed reaches 0.862 Acc and 0.805 AUROC, surpassing Large-Ensemble by about 3.1% AUROC.
- On MIMIC-IV-MM longstay, AdaCoMed outperforms the best baseline MultimodalPred-Small by 6.1% AUROC and 3.4% mF1.
- On MMIST ccRCC 12-month survival (Vital-12), AdaCoMed (Threshold Contrastive) achieves 0.898 AUROC versus 0.778 for Large-Ensemble, with 0.628 mF1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| collaborate large and small models with AdaCoMed | MIMIC-IV-MM · mortality | Auroc: 0.774 -> 0.805 | Helped |
| collaborate large and small models with AdaCoMed | MIMIC-IV-MM · longstay | Auroc: 0.690 -> 0.732 | Helped |
| collaborate large and small models with AdaCoMed | MIMIC-IV-MM · readmission | Auroc: 0.592 -> 0.619 | Helped |
| collaborate large and small models with AdaCoMed | MMIST ccRCC · Vital-12 | Auroc: 0.778 -> 0.898 | Helped |
| use learnable weighted fusion (ablation) | AdaCoMed · MIMIC-IV-MM mortality | Auroc: 0.805 -> 0.786 | Hurt |
| use inverse contrastive weighting (ablation) | AdaCoMed · MIMIC-IV-MM longstay | Auroc: 0.732 -> 0.718 | Hurt |
| use Gaussian contrastive weighting (ablation) | AdaCoMed · MIMIC-IV-MM longstay | Auroc: 0.718 -> 0.732 | Helped |
| use threshold contrastive weighting (ablation) | AdaCoMed · MIMIC-IV-MM readmission | Auroc: 0.637 -> 0.601 | Hurt |
| use inverse contrastive weighting (ablation) | AdaCoMed · MIMIC-IV-MM readmission | Auroc: 0.628 -> 0.637 | Helped |
| use Gaussian contrastive weighting (ablation) | AdaCoMed · MMIST ccRCC Vital-12 | Auroc: 0.847 -> 0.856 | Helped |
| use threshold contrastive weighting (ablation) | AdaCoMed · MMIST ccRCC Vital-12 | Auroc: 0.856 -> 0.898 | Helped |
| vary the contrastive-loss weight λ (ablation) | AdaCoMed · MIMIC-IV-MM mortality and readmission | Auroc | Hurt |
Surprising findings
- Naive fusion of single-modal large models leads to modality domination, where pretrained encoder capability rather than clinical importance determines modality contribution.
- Some baselines (e.g., WSI-Large, CT-Large) reach higher accuracy on MMIST only through prediction collapse caused by label imbalance.
Limitations
- Evaluated on only two medical datasets and four diagnostic tasks.
- Adaptive weighting strategies require hyperparameter choice (temperature, scaling, threshold, slope) and their best option varies by task.
- Sensitivity to the contrastive loss weight lambda differs across tasks (mortality and readmission degrade at larger values).
Experimental setup
Datasets: MIMIC-IV-MM MIMIC-CXR-JPG MIMIC-IV-Note MIMIC-IV MMIST ccRCC
Benchmarks: MIMIC-IV-MM MMIST ccRCC
Metrics: Accuracy AUROC AUPRC mAP mAR mF1
Baselines: Meta-Transformer OneLLM XRay-Large Text-Large TimeSeries-Large Large-Ensemble MultimodalPred-Small WSI-Large CT-Large Clinical-Large
Builds on: Meta-Transformer MultimodalPred Mixture of Experts CLAM
Closest papers by meaning
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.88)
- MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis (CVPR 2026 · similarity 0.88)
- RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis (NeurIPS 2025 · similarity 0.86)
- MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.86)
- MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow (ICLR 2026 · similarity 0.86)
- DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.