Research map / Vision-language and multimodal / Cross-modal learning
HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HEALNet, a hybrid early-fusion iterative cross-attention network that fuses WSI and multi-omic data through a shared latent array with modality-specific attention, evaluated on TCGA survival analysis.
Problem
Multimodal fusion of structurally heterogeneous biomedical data (histopathology images, tabular omics) where existing early, intermediate, and late fusion methods either discard structure, explode dimensionality, cannot capture cross-modal interactions, or handle missing modalities poorly.
Approach
A shared latent bottleneck array is iteratively updated via modality-specific cross-attention over raw inputs (with weight sharing across layers), so cross-modal interactions are learned early while preserving each modality's structure; missing modalities are handled by simply skipping their update step.
Key results
- On TCGA survival, HEALNet reaches c-Index 0.668 (BLCA), 0.638 (BRCA), 0.812 (KIRP), best among multimodal baselines on three of four sites; UCEC 0.626 is comparable to Perceiver 0.623 and unimodal WSI 0.630.
- Average multimodal uplift of up to 7% over the best unimodal baseline and up to 4.5% over the best early, intermediate and late fusion baselines.
- On MIMIC-III, HEALNet achieves ICD9 AUC 0.767 and MORT macro-AUC 0.748, above Perceiver (0.733, 0.723).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HEALNet multimodal fusion instead of the strongest multimodal baseline | HEALNet · BLCA | c-Index: 0.631 -> 0.668 | Helped |
| use HEALNet multimodal fusion instead of the strongest multimodal baseline | HEALNet · BRCA | c-Index: 0.630 -> 0.638 | Helped |
| use HEALNet multimodal fusion instead of the strongest multimodal baseline | HEALNet · KIRP | c-Index: 0.810 -> 0.812 | Helped |
| use HEALNet multimodal fusion instead of the strongest multimodal baseline | HEALNet · UCEC | c-Index: 0.623 -> 0.626 | Helped |
| use HEALNet multimodal fusion instead of the strongest unimodal baseline | HEALNet · BLCA | c-Index: 0.606 -> 0.668 | Helped |
| use HEALNet multimodal fusion instead of the strongest unimodal baseline | HEALNet · BRCA | c-Index: 0.580 -> 0.638 | Helped |
| use HEALNet multimodal fusion instead of the strongest unimodal baseline | HEALNet · KIRP | c-Index: 0.780 -> 0.812 | Helped |
| use HEALNet multimodal fusion instead of the strongest unimodal baseline | HEALNet · UCEC | c-Index: 0.630 -> 0.626 | No clear effect |
| use HEALNet multimodal training with inference on only Omics | HEALNet · BLCA | c-Index: 0.606 -> 0.618 | Helped |
| use HEALNet multimodal training with inference on only WSI | HEALNet · BLCA | c-Index: 0.487 -> 0.501 | Helped |
| use HEALNet multimodal training when half of test samples have WSI and half have Omics | HEALNet · BRCA | c-Index: 0.543 -> 0.541 | No clear effect |
| use HEALNet multimodal training when half of test samples have WSI and half have Omics | HEALNet · BLCA | c-Index: 0.547 -> 0.612 | Helped |
Surprising findings
- On UCEC, where WSI alone is informative, HEALNet's omic attention weights barely changed from initialisation, suggesting it learned to inhibit the uninformative modality.
Limitations
- Sensitive to regularisation choice and prone to overfitting given many attention parameters on few samples.
- Evaluated only on survival analysis with TCGA cancer datasets of a few hundred to ~1000 patients; not validated clinically.
- UCEC shows modality dominance where multimodal gains are absent, so HEALNet only matches the best unimodal model there.
Experimental setup
Datasets: TCGA-BLCA TCGA-BRCA TCGA-KIRP TCGA-UCEC MIMIC-III Kather100K MoNuSAC
Benchmarks: TCGA-BLCA TCGA-BRCA TCGA-KIRP TCGA-UCEC MIMIC-III
Metrics: c-index AUC macro-AUC
Baselines: Porpoise MCAT MOTCAT MultiModN Perceiver AMIL unimodal omic and WSI models
Builds on: Perceiver MCAT Porpoise MOTCAT self-normalising neural networks
Closest papers by meaning
- Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine (ICLR 2025 · similarity 0.88)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.87)
- Multimodal Prototyping for cancer survival prediction (ICML 2024 · similarity 0.87)
- MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine (ICLR 2025 · similarity 0.87)
- When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Framework (NeurIPS 2026 · similarity 0.86)
- Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.