Research map / Vision-language and multimodal / Cross-modal learning
DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DrFuse, which disentangles shared and modality-distinct representations of EHR and chest X-rays and uses a disease-aware attention fusion with an attention ranking loss to handle missing CXR and modal inconsistency in clinical phenotype prediction.
Problem
Clinical multi-modal fusion of EHR time series and chest X-rays suffers from missing modalities (many patients lack CXR) and modal inconsistency, where each modality's predictive significance varies by patient and disease.
Approach
Disentangle shared and distinct features per modality, align shared features via JSD with a logit-pooling fusion, impose orthogonality constraints, and fuse modalities with per-disease attention trained by an auxiliary-loss-driven attention ranking loss.
Key results
- On the matched subset (trained and tested on matched), DrFuse macro PRAUC 0.450 vs MedFuse 0.427, about 5.4% relative improvement.
- Trained on full dataset and tested on matched subset, DrFuse reaches 0.470 PRAUC vs MedFuse 0.434.
- Per-disease average rank of 2.04 for DrFuse, best among compared methods (MedFuse 2.68, Transformer 3.24, ResNet50 3.96).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DrFuse instead of MedFuse | Clinical EHR-CXR phenotype prediction · MIMIC-IV/MIMIC-CXR matched subset, trained on matched subset | PRAUC: 0.427 (0.387, 0.473) -> 0.450 (0.426, 0.498) | Helped |
| use DrFuse instead of Transformer | Clinical EHR-CXR phenotype prediction · MIMIC-IV/MIMIC-CXR full dataset, trained on matched subset | PRAUC: 0.374 (0.355, 0.395) -> 0.384 (0.371, 0.402) | Helped |
| use DrFuse instead of Transformer | Clinical EHR-CXR phenotype prediction · MIMIC-IV/MIMIC-CXR matched subset, trained on full dataset | PRAUC: 0.435 (0.393, 0.481) -> 0.470 (0.420, 0.512) | Helped |
| use DrFuse instead of Transformer | Clinical EHR-CXR phenotype prediction · MIMIC-IV/MIMIC-CXR full dataset, trained on full dataset | PRAUC: 0.418 (0.398, 0.440) -> 0.419 (0.391, 0.434) | Helped |
| add JSD alignment and orthogonality losses (ablation) | DrFuse · MIMIC-IV/MIMIC-CXR matched subset | PRAUC: 0.446 (0.411, 0.501) -> 0.450 (0.426, 0.498) | Helped |
| add JSD alignment and orthogonality losses (ablation) | DrFuse · MIMIC-IV/MIMIC-CXR full dataset | PRAUC: 0.374 (0.355, 0.395) -> 0.384 (0.371, 0.402) | Helped |
| replace JSD with MSE and logit pooling with average pooling (ablation) | DrFuse · MIMIC-IV/MIMIC-CXR matched subset | PRAUC: 0.447 (0.410, 0.498) -> 0.450 (0.426, 0.498) | Helped |
| replace JSD with MSE and logit pooling with average pooling (ablation) | DrFuse · MIMIC-IV/MIMIC-CXR full dataset | PRAUC: 0.375 (0.356, 0.396) -> 0.384 (0.371, 0.402) | Helped |
| add attention ranking loss (ablation) | DrFuse · MIMIC-IV/MIMIC-CXR matched subset | PRAUC: 0.438 (0.396, 0.485) -> 0.450 (0.426, 0.498) | Helped |
| add attention ranking loss (ablation) | DrFuse · MIMIC-IV/MIMIC-CXR full dataset | PRAUC: 0.361 (0.343, 0.382) -> 0.384 (0.371, 0.402) | Helped |
Surprising findings
- Multi-modal fusion does not always help: MedFuse drops 40.5% on conduction disorders relative to the best uni-modal prediction, illustrating modal inconsistency.
- All methods, including the uni-modal Transformer, score lower on the full test set than on the matched subset, suggesting domain shift between patients with and without CXR.
Limitations
- Domain shift between patients with and without CXR limits performance on the full dataset.
- Evaluated on a single pair of datasets (MIMIC-IV and MIMIC-CXR) and one task family (phenotype prediction).
Experimental setup
Datasets: MIMIC-IV MIMIC-CXR
Benchmarks: MIMIC-IV phenotype classification (25 labels)
Metrics: PRAUC macro-averaged PRAUC average rank
Baselines: Transformer MMTM DAFT MedFuse MedFuse-II
Builds on: MedFuse MMTM DAFT Transformer
Closest papers by meaning
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.87)
- Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation (NeurIPS 2024 · similarity 0.86)
- Multi-modal Medical Diagnosis via Large-small Model Collaboration (CVPR 2025 · similarity 0.86)
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.85)
- HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data (NeurIPS 2024 · similarity 0.85)
- Incomplete Modality Disentangled Representation for Ophthalmic Disease Grading and Diagnosis (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.